NeRF Compression via Transform Coding

We thank the reviewer for your constructive comments and for the recognition of our neural compression approach for NeRFs. We answer each of the reviewer's concerns below:

Response to the Weakness section:

The improvement over the previous method is not so significant, the TC-TensoRF-L only shows notable improvement in the LLFF dataset, while in other datasets, it only brings minor performance gain in both visual quality（PSNR）and storage size. Though it provides a trade-off curve against VQ-TensoRF, the author only changed the codebook size of VQ-TensoRF, which may indicate that the curve was not drawn on the optimal hyper-parameter for VQ-TensoRF.

In our experiments, our method demonstrates consistent improvements over VQ-TensoRF across four datasets. Given the high baseline performance of current methodologies and the maximum achievable quality of uncompressed models, we consider the enhancements in both reconstruction quality and storage efficiency achieved by our method to be substantial.

In our rate-distortion curve experiments, the aim was to evaluate the performance of our method against VQ-TensoRF under various compression levels. To achieve this, we only adjusted the codebook size hyperparameter for VQ-TensoRF and the rate-distortion hyperparameter λ for our model. We believe our setting is fair for both methods. While additional tuning of hyperparameters might improve the compression capabilities of both models, we believe that the extensive effort required for such optimization is not feasible for practical, real-world use of NeRF compression.

Though the author has discussed DVGO and Plenoxels and tends to treat applying compression to those methods as a future work, I believe this could be a major weakness to not showing that the proposed is capable of generalize to other grid-based methods. As those methods have different design methodology e.g. plenoxels is already sparse and do not rely on MLP to recover feature in grid point. and it is important to show the proposed transform coding can still work for different types of grid-based nerf.

We acknowledge that currently our method is specifically designed for feature plane-based NeRF, which may seem more restrictive compared to other methods. However, we believe that applying neural compression to NeRF is an interesting direction, and our work provides valuable insights and sets a foundation for future extensions of neural compression to other NeRF architectures.

Response to the questions:

what is the final composition of the storage size? It would be better to display the composition in a chart or figure to help others better understand the proposed method.

The average storage size breakdown of our model on the NeRF Synthetic dataset is provided in the following table. For the feature planes, we compressed it with the learned entropy model. All the other components (the renderer network, the decoder, density/appearance vectors, learned mask, entropy bottleneck parameters and model config) are packed into a single file and compressed with LZ77 using numpy.

We thank the reviewer for your constructive comments and for the recognition of our neural compression approach for NeRFs. We answer each of the reviewer's concerns below:

Response to the Weakness section:

Compared with existing work like VQ-TensoRF, the performance gain of the proposed method under a similar compression ratio is marginal in some cases (for low compression). Besides, this work is currently only designed for feature plane-based NeRF methods while other compared methods are more generalized.

Our method actually demonstrates consistent improvements over VQ-TensoRF across four datasets. Notably, in the LLFF dataset, our model achieves a similar level of reconstruction quality to VQ-TensoRF while requiring only half the model size. Additionally, our rate-distortion curve outperforms that of VQ-TensoRF.

We acknowledge that currently our method is specifically designed for feature plane-based NeRF, which may seem more restrictive compared to other methods. However, we believe that applying neural compression to NeRF is an interesting direction, and our work provides valuable insights and sets a foundation for future extensions of neural compression to other NeRF architectures.

The training and rendering time, especially the time needed for compressing and decompressing the features should be provided and compared with previous works.

For the training time comparison: Training our compression model on a scene from the NeRF-Synthetic dataset takes around 40 minutes on NVIDIA RTX A6000. The baseline VQ-TensoRF is faster and takes around 7 minutes on the same machine. For the baseline Re:TensoRF, we do not have the official code and can not measure the training time.

At rendering time, our decompression procedure only takes 2 seconds more. VQ-TensoRF extraction takes around 0.02 second.

The overall contribution of this paper is not sufficient enough. The transform coding framework is also similar to existing frameworks in image and video compression.

While our framework adapts the traditional transform coding methods used in image and video compression, it is distinct in our application of transform coding to NeRF. This application presents unique challenges and requirements, distinct from conventional images and videos. Our approach adapts and extends the principles of transform coding to address these specific challenges in 3D scene representation and compression. To further enhance the performance of our framework for NeRF, we proposed a unique framework with the omission of the encoder, a lightweight decoder; combined with an importance-weighted reconstruction loss and a masked entropy model. We believe our work also serves as a bridge, bringing well-established techniques from the realm of 2D media compression into the emerging field of 3D scene reconstruction. This not only provides a new perspective on familiar techniques but also opens up avenues for further research and development in NeRF compression.

Response to the questions:

The author proposed the latent decoder without an encoder. It will better demonstrate the efficiency by also showing the performance of a compressor with both an encoder and decoder during training and then drop the encoder during inference.

We show the comparison below for the NeRF-Synthetic dataset with our low compression architecture. Removing the encoder helped improve the compression performance. Note that this result is from one of our preliminary experiments, and we did not use the masked entropy model at that time.

References:

[1] Compressing Volumetric Radiance Fields to 1 MB. CVPR 2023.

[2] Learning End-to-End Lossy Image Compression: A Benchmark. TPAMI 2022.

[3] An Introduction to Neural Data Compression. Foundations and Trends in Computer Graphics and Vision.

It would be beneficial to include a comparison of the training time between the proposed method and other established methods. Additionally, the paper should provide a more precise breakdown of the storage sizes for each component, including the decoder, entropy model, binary mask, and grid feature.

For the training time comparison: Training our compression model on a scene from the NeRF-Synthetic dataset takes around 40 minutes on NVIDIA RTX A6000. The baseline VQ-TensoRF is faster and takes around 7 minutes on the same machine. For the baseline Re:TensoRF, we do not have the official code and can not measure the training time.

The average storage size breakdown of our model on the NeRF Synthetic dataset is provided in the following table. For the feature planes, we compressed it with the learned entropy model. All the other components (the renderer network, the decoder, density/appearance vectors, learned mask, entropy bottleneck parameters and model config) are packed into a single file and compressed with LZ77 using numpy.

The term 'transform coding' in the paper's title may not be entirely appropriate, as the paper employs an encoder-free framework that directly learns the latent code without involving analysis transforms within the coding paradigm

The term 'transform coding' in our title was selected to broadly encompass the conceptual essence of our approach, which involves transforming data into a more compressible format. While our paper indeed introduces an encoder-free framework that directly learns the latent code, we believe this process aligns with the fundamental principles of transform coding. The direct learning of the latent code can be viewed as an implicit form of analysis transform, where the input data is converted into a representation that is more amenable to efficient compression.

However, we acknowledge that the term 'transform coding' might traditionally be associated with specific technical implementations involving explicit analysis and synthesis transforms. In light of your feedback, we are open to considering alternative names that may more precisely communicate the specifics of our methodology.

Response to the questions:

What are the advantages of the proposed compression method when compared to existing pruning or vector quantization methods?

Neural compression is a well-established method for compressing data with a strong theoretical foundation, and can yield state-of-the-art performance on images and videos [3]. On the other hand, we believe that vector quantization has some disadvantages, such as the need of a regularization to fully utilize the codebook [1]. We also empirically found that the neural compression works better than vector quantization and pruning in our experiments. Understanding the fundamental differences between the two frameworks in compression could be an interesting future work.

Have the authors considered the results of incorporating an encoder and training the entire network end-to-end?

We considered training with an encoder. We show the comparison below for the NeRF-Synthetic dataset with our low compression architecture. Removing the encoder helped improve the compression performance. Note that this result is from one of our preliminary experiments, and we did not use the masked entropy model at that time.

It is recommended that the authors provide a detailed breakdown of the storage size for each component to offer a more comprehensive understanding.

We provided the breakdown of the storage size in our response to the Weakness section.

In Figure 5, I noticed that the blue curve (TC-TensoRF-L) is situated below the orange (w/ Factorized Prior) and green (w/o Importance Weight) curves in the lower bitrate range. Could you please provide an explanation for this observation?

In the case of the Factorized Prior model, we hypothesize that at lower bitrate, the cost of sending the masks is more than the feature planes size saved when using the masked entropy model. Whereas in the comparison with the model not using the importance weight mechanism, at lower bitrate, TC-TensoRF-L is forced to completely ignore locations with low to medium importance, leading to lower reconstruction quality overall.

We thank the reviewer for your constructive comments and for the recognition of our unique approach compared to existing compression methods. We answer each of the reviewer's concerns below:

Response to the Weakness section:

While the introduction of a neural compression method to encode the grid representation has demonstrated superior RD performance, the underlying motivation for this approach is not adequately elucidated.

Our underlying motivation for using neural compression is grounded in its proven efficacy in image compression, where it has consistently demonstrated superior rate-distortion (R-D) performance compared to traditional compression methods. Its ability to efficiently manage high-dimensional data make it particularly suitable for handling the complexities associated with grid representations in NeRFs. Upon examination of grid-based methods such as TensoRF, we observed that the grid features often exhibit significant redundancies and can be effectively compressed with neural compression.

The novelty of this paper is not sufficient. The introduction of the weighted-rate-distortion loss lacks significant new contributions, as it heavily relies on an existing importance score calculation method [1]. The proposed binary mask entropy coding resembles the concept of importance map-based bit allocation schemes, such as in [2]. Moreover, the compression method falls short in considering contextual information for further reduction of spatial redundancy.

In response to the concerns of The Reviewer:

1. Adaptation of Importance Weighted Loss: We agree that our approach is based on an established importance score calculation method. However, our unique application of this score as a training weight in our model represents a contribution. Unlike previous works [1] that applied this score with a pruning threshold and required hyperparameters tuning, our method integrates this score with the rate-distortion loss commonly used in neural compression. This integration enables our model to adeptly prioritize critical grid locations across varying compression levels λ.

2. Binary Mask Entropy Coding in NeRFs: While there are conceptual similarities with existing importance map-based bit allocation techniques, our application in the realm of NeRFs is notably distinct. Our strategy leverages the sparsity characteristic of NeRF representations to optimize the encoding process, setting our approach apart from traditional methods used for images. Additionally, our use of Gumbel-Softmax for automatic learning of the mask is a unique aspect that differentiates our method from prior works.

3. Contextual Information for Spatial Redundancy Reduction: We acknowledge the point raised about the potential for further reducing spatial redundancy through contextual information. This presents an opportunity for future work and improvement.

4. In our paper, we also proposed several modifications to the existing neural compression framework. These include the omission of the encoder and the implementation of a light-weighted decoder. While we recognize that these modifications may not be groundbreaking in their novelty, we believe that they play a crucial role in enhancing the overall efficiency and effectiveness of our framework.