NeCGS: Neural Compression for 3D Geometry Sets

Comment 1. The manuscript ... more practical.

Response:

• Thanks for the valuable suggestion. First, we clarify the optimization process of converting 3D models into TSDF-Def 4D volumes is efficient, as shown in the table below, while 15 hours refers to the time consumed by the whole compression process, including the TSDF-Def representation process and optimization process of both auto-decoder and features.
• The optimization process of converting the 3D models of a dataset into TSDF-Def volumes can be parallelized using multiple GPUs. In our experiment, we utilized 8 NVIDIA RTX 3090 GPUs.The table below details the time consumed for processing the mixed dataset. |Resolution| Time per Shape (s)|Total Time (h)| |-----|-----|-----| |32|22.39 | 0.46 | |48| 24.34| 0.50| |64| 26.67| 0.55 | |128| 28.03| 0.58| |256|41.51 |0.86 |
• Second, we want to emphasize that our NecGS is designed for the offline compression of 3D geometry datasets to save storage space, where the optimization/compression time should not be a key factor. Instead, we should concentrate more on the decompression/decoding speed because the users expect to obtain the 3D models timely when querying the dataset. As shown in Table 3, when the resolution is 128, the decompression time of our NeCGS is only 98.95 ms, satisfying the real-time requirement.
• Moreover, there are various potential solutions to accelerate the optimization. At the software level, more efficient convolution (e.g., using multiple 1-D or 2-D convolution to approximate the 3-D convolution or using 3D sparse convolution to replace the traditional 3D convolution) can be used to speed up the process. At the hardware level, optimization programs can be run using multiple GPUs or other more efficient hardware. Like NeRF, the initial algorithms required several days for optimization. Subsequently, improved methods have been introduced to accelerate the optimization process to a few minutes or even seconds.

Comment 2. While the method ... open boundaries (cloth).

Response:

• Our NeCGS can be adapted to newly added 3D models to the dataset. Given a new 3D model, we first represent it into a TSDF-Def volume through Algorithm 1. Then following the optimization progress in Sec. 3.2, we only optimize its corresponding embedded feature while keeping the trained decoder unchanged. The visual results are shown in Fig. R1 of the uploaded PDF, demonstrating its generalization ability to new geometry data. Moreover, in this situation, the optimization is significantly less because only embedded features are optimized.
• The dataset we used indeed consists of complex shapes, as shown in Fig. 1, where the complex structures of the decompressed models remain. In Fig. R5 of the uploaded PDF files, we demonstrate additional complex shapes.
• The signs in TSDF are ambiguous on models with open boundaries, our method cannot directly compress models of this category. However, an alternative and straightforward approach is to utilize UDF (unsigned distance field) rather than SDF to represent them in our method, which allows for the processing of models with open boundaries.

Comment 3. The choice of an ... over others.

Response:
In the ablation study, we indeed compared our auto-decoder framework with auto-encoder, a widely used framework. The visual results shown in Fig. 8(a) demonstrate the superiority of our decoder-based structure. In the auto-encoder framework, the embedded features are adjusted by optimizing the encoder, which is less flexible than the auto-decoder, where the embedded features are optimized directly. In the final version, we will provide more explanations.

Comment 4. While the ..., would be insightful.

Response:
We refer the reviewer to the 4th response to Reviewer qNSe for comparison of compression time.

Comment 5. The paper ... complex datasets.

• Thank you for the insightful comments. In the final version, we will complement more discussions about this point. In our experiments, the embedded features and the decoder are optimized over a fixed number of epochs with a constant batch size. Consequently, the overall optimization time and computational expenses scale proportionally with the amount of geometric shapes being compressed.
• To validate this, in addition to the mixed dataset utilized in the experiments (600 shapes), we create two additional mixed datasets of different sizes by selecting 100 and 300 shapes from the remaining three datasets. This results in mixed datasets comprising 300 and 900 shapes, respectively. The table below shows the optimization times for various Mixed datasets. |# Shapes|Optimization Time (h)| |-----|-----| |300|8.25| |600|16.32| |900|24.37|
• More importantly, we also want to emphasize that our NeCGS our NecGS is designed for the offline compression of 3D geometry datasets to save storage space, where the optimization/compression time should not be a key factor. Instead, we should concentrate more on the decompression/decoding speed because the users expect to obtain the 3D models timely when querying the dataset. As shown in Table 3, when the resolution is 128, the decompression time of our NeCGS is only 98.95 ms, satisfying the real-time requirement.
• Besides, there are various potential solutions to accelerate the optimization. At the software level, more efficient convolution (e.g., using multiple 1-D or 2-D convolution to approximate the 3-D convolution or using 3D sparse convolution to replace the traditional 3D convolution) can be used to speed up the process. At the hardware level, optimization programs can be run using multiple GPUs or other more efficient hardware. Like NeRF, the initial algorithms required several days for optimization. Subsequently, improved methods have been introduced to accelerate the optimization process to a few minutes or even seconds.

Dear Reviewer 18DA

Thanks for your time and effort in reviewing our manuscript and the favorable recommendation. In our previous response, we addressed your remaining concerns directly and comprehensively. We very much look forward to your further feedback on our responses.

Best regards,

The authors

Comment 1. The reconstruction results ... baseline methods?

Response: * Actually, when zooming in Fig. 5 of the manuscript, the decompressed shapes by our NeCGS exhibit superior quality to those by GPCC, showcasing significantly smoother shapes decompressed by NeCGS.

• We refer the reviewer to the 4th response to Reviewer qNSe for comparison of compression time.

Comment 2. In the ablation ... significantly deteriorates?

Response: * The 3D models reconstructed from their much low-resolution TSDF-Def volumes will exhibit substantial errors while leading to a notable increase in compression distortion.

• In addition to the resolution examined in the ablation study, we also experimented with lower resolutions, specifically 32 and 48. The distortions are presented in the table below. It is evident that decreasing the resolution from 64 to 48 results in a significant increase in distortion. Visual representations can be found in Fig. R2 of the uploaded PDF file.
  |Res.|Size (MB)|Com. Ratio| CD (1e-3) | NC| F1-0.005|F1-0.01| |-----|-----|-----|-----|-----|-----|-----| |32|1.037|364.898|16.982|0.872|0.215|0.542| |48|1.269|298.187|12.240|0.895|0.314|0.705| |64|1.408|268.75|4.271|0.927|0.721|0.966| |128|1.493|253.45|3.436|0.952|0.842|0.991| |256|1.627|232.58|3.234|0.962|0.870|0.995|

Comment 3. What happens ... or significantly lesser?

Response: Our NeCGS can be adapted to newly added 3D models to the dataset. Given a new 3D model, we first represent it into a TSDF-Def volume through Algorithm 1. Then following the optimization progress in Sec. 3.2, we only optimize its corresponding embedded feature while keeping the trained decoder unchanged. The visual results are shown in Fig. R1 of the uploaded PDF, demonstrating its generalization ability to new geometry data. Moreover, it is worth noting that in this situation, the optimization is significantly less because only embedded features are optimized.

Comment 4. It would be interesting ... geometry dataset?

Response: * In the experiment, the embedded features and the decoder are optimized over a fixed number of epochs with a constant batch size. Consequently, the overall optimization time and computational expenses scale proportionally with the amount of geometric shapes being compressed.

• To validate this, in addition to the mixed dataset utilized in the experiments (600 shapes), we create two additional mixed datasets of different sizes by selecting 100 and 300 shapes from the remaining three datasets. This results in mixed datasets comprising 300 and 900 shapes, respectively. The table below displays the optimization times for various Mixed datasets. |# Shapes|Optimization Time (h)| |-----|-----| |300|8.25| |600|16.32| |900|24.37|

Comment 5. Are the latent vectors ... regarding that.

Response: Before the optimization, the latent vectors (embedded features) are initialized as random Gaussian noise with a mean of 0 and a standard deviation of 1/3. In the experiment, with the quantization limits set at 1 and -1, the 1/3 standard deviation ensures that nearly every value of the initialized latent vectors falls within the [-1, 1] range, aligning with the three-sigma rule of Gaussian distribution.

Comment 6. More details ... in the paper.

Response: *Our DMC is modified from Marching Cubes and used to extract surfaces from TSDF-Def volumes. Besides TSDF, we assign a deformation for each corner of the cubes, making the cubes adjust detailed structures of the shapes, as shown in Fig. 3. The triangle extraction in each cube is the same as the original Marching Cubes, where the only difference is the coordinates of cube corners. We will add more detailed description of DMC in the future version.

• Algorithm 1 summarizes the whole optimization process to optimize the TSDF-Def volume from the given 3D model. Given 3D model, $\mathbf{S}$, we can optimize and obtain its corresponding TSDF-Def volume $\mathbf{V}$ through Algorithm 1. Initially, we distribute grid points $\mathbf{G}$ uniformly across space, serving as the corners of the cubes utilized in DMC. Before optimization, we initialize $\mathbf{V}[...,0]$ as the ground truth TSDF at the location of $\mathbf{G}$ and the deformation as $\mathbf{V}[...,1:3]=0$. During the optimization, the optimal TSDF-Def volume could be optimized by minimizing the difference between the reconstructed shape $`DMC`(\mathbf{V})$ and the original shape $\mathbf{S}$. In the future version, we will clarify the unclear descriptions to make the optimization process easier to understand.

Comment 7. In Fig. 4, is ... increase it?

Response: * The compression ratio is highly related to the reconstruction error, i.e., a larger compression ratio generally introduces more serious reconstruction errors. Only discussing the compression ratio does not make much sense. In practice, we need to balance compression ratio and reconstruction accuracy according to requirements.\ *To answer your question, We further increase the compression ratio, and the quantitative results on the Mixed dataset are shown in Fig. R4 of the uploaded PDF file. Obviously, when increasing the compression ratio, our NeCGS is still better than baseline methods.

Comment 8. Figure 6 is before 5 in the paper.

Response: Thanks. We will carefully check the layout.

Dear Reviewer jc1U

Thanks for your time and effort in reviewing our manuscript and the favorable recommendation. In our previous response, we addressed your remaining concerns directly and comprehensively. We very much look forward to your further feedback on our responses.

Best regards,

The authors

It is great to receive your further feedback, showing that our initial responses have addressed many of your concerns. In the following, we will address your remaining questions.

1. From the results it seems that to achieve a reasonable accuracy, a resolution of 128 or 256 is required and the optimization requires about 24 hrs for a dataset containing just 600 geometries (roughly 400Mb).

• As demonstrated in Table 3 of our manuscript, a resolution of 128 is sufficiently precise for the reconstructed meshes, requiring approximately 16 hours to finalize the optimization process.

• In the response, we have gathered the compression times of different methods. It is worth noting that VPCC necessitates around 40 hours to finalize compression, which is much more time-consuming than our method. Additionally, the training process for method PCGCv2 is also time-consuming (PCGCv2 requires numerous hours for training. In the Table of the initial response, the time only counts the inference time without considering the training time.). Besides, we have explored multiple techniques to expedite the optimization procedure.

2. The ability to add new samples to the existing dataset without spending much ... training distribution. Can you provide the error curves of this optimization to verify that these losses are actually decreasing and it is being able to find the correct embedded vectors representing this geometry? Would it be possible to add more diverse geometries to this experiment and report the reconstruction accuracy in each case? How does GPCC perform for the same geometries?

• The table below displays the reconstruction accuracy for unseen meshes during optimization. Through iterative processes, the precision of the reconstructed model is steadily enhanced.

• Thingi10K comprises a total of 10,000 distinct meshes. Consequently, the unseen meshes sourced from the Thingi10K dataset exhibit greater diversity, with their reconstructed outcome depicted in Figure R1 of the provided PDF.

• We evaluate the accuracy of the generalized new meshes, as illustrated in the table below. It is evident that the reconstructed new meshes exhibit greater errors compared to the training meshes. Nonetheless, our method excels in reconstructing the overall shapes and decompressing unseen meshes more accurately than GPCC.

3. *Now, if we consider datasets containing 60000 geometries where compression is truly required this method becomes computationally prohibitive and has no utility. *

• The simplest and most straightforward approach involves grouping the samples of the dataset, compressing each group separately in parallel, thereby cutting down on compression time.

• By solely optimizing the embedded features and keeping the decoder weights fixed for new meshes, the average optimization time per new mesh significantly reduces, offering a fresh approach to compressing extensive geometric data. To achieve this, 1) Initially, a small subset of data is chosen for optimizing the embedded features and decoder weights. 2) Subsequently, the decoder weights are set, and only the embedded features are optimized for the remaining meshes, enabling rough reconstruction post-optimization. 3) The embedded features of all meshes are refined, along with the decoder weights, over several epochs. This three-stage optimization strategy, as opposed to directly optimizing all embedded features and decoder weights, results in considerable time savings.

Comment 1. The intuition behind preferring TSDF-Def 4D volume over TSDF 3D volume is unclear, ... The quantitative results in Table 2 only show marginal improvements. An brief intuitive explanation of the design choice is helpful.

Response:

• For a geometry dataset usually containing 3D models with various simple and complex structures, TSDF can represent 3D models with simple structures but fails to accurately represent them with complex ones when the resolution is relatively limited. Differently, we propose TSDF-Def 4D volume by assigning additional offsets at each corner of the cubes, so that it can well represent 3D models with both simple and complex structures.
• As Fig.7 shows, compared to TSDF volume, our TSDF-Def volume can handle thin structures of the shapes and has demonstrated much better performance. In Table 2, the quantitative advantage of our TSDF-Def over TSDF is marginal because the results are the average error of the entire dataset.
• To address your concern, we divide the dataset into two parts when evaluating compression performance: (1) the shapes with thin and fine structures (20 shapes); and (2) those without detailed structures.
• The quantitative results are shown in the following table, where it can be seen that our TSDF-Def volumes can significantly reduce the distortion for the models with thing structures. It is obvious that the advantage of our TSDF-Def 4D volume would be more significant than the TSDF 3D volume on the datasets with more complex and fine structures. Fig. R3 demonstrates the selected models. |Representation|Data|CD(1E-3)|NC|F1-0.005|F1-0.01| |----|----|----|----|----|----| |TSDF|All shapes|5.015|0.944|0.662|0.936| |TSDF|Shapes w/ thin structures|11.628|0.454|0.728|0.861| |TSDF|Shapes w/o thing structures|4.786|0.670|0.943|0.947| |TSDF-Def|All shapes|4.913|0.947|0.674|0.943| |TSDF-Def|Shapes w/ thin structures|8.751|0.506|0.800|0.874| |TSDF-Def|Shapes w/o thing structures|4.783|0.680|0.948|0.950|

Comment 2. There are lot of methods which try to compress a neural field. For e.g. Triplanes[1], HashGrid [2], Vector Quantization [3], TensoRF [4], Dictionary Fields [5]. It is not very clear why this method does not compare with all these techniques which can be used for compression?

Response:
The methods in [1-5] are focused on the compact representation of feature volumes, where the continuous feature of any point in the space could be obtained through bilinear or trilinear interpolation. Although these methods could be used to compress geometry models, the required storage space can be as high as several MBs, as shown in Fig.7 of [2], Fig.5 of [5], and so on, which is much larger than our method (averaging only a few KBs per model). When reducing the feature dimension to make the feature to be KBs, these methods fail to recover shapes from their implicit fields. We believe that these algorithms can be combined with some compression techniques and additional efforts in the future to achieve more efficient compression. For these reasons, our current baseline methods do not include these methods.

Comment 3. Can this method generalize? Can I use the trained auto-decoder setting to compress a new 3D mesh on which the model is not trained on? How about other methods with which the method compares.

Response:

• Thank you for your insightful comment! Yes, our NeCGS can be generalized to new 3D models. Specifically, given a new 3D model, we first represent it into a TSDF-Def 4D volume through Algorithm 1. Then following the optimization progress in Sec. 3.2, we only optimize its corresponding embedded feature while keeping the trained decoder unchanged.
• The visual results are shown in Fig. R1 of the uploaded PDF, showing the ability to generalize. It is not surprising that our NeCGS can be generalized to new 3D models because after fitting 3D models with various structures, the decoder can learn the prior knowledge of various geometry data, and it can represent unseen models by only optimizing its embedded features. The methods under comparison can also be adapted to new 3D models.
• GPCC, VPCC, and Draco are traditional compression methods that do not require training, enabling them to compress new meshes. PCGCv2 could utilizes the trained encoder to compress the new models directly. QuantDeepSDF operates as an auto-decoder framework, allowing it to generalize new models similarly to our NeCGS.

Comment 4. The paper does not do a relative comparison of the compression time with the baseline methods. Given ..., I have concerns about the practical usage of this method.

Response:

• As shown in the table below, compared to compression methods for single shapes, i.e., GPCC, PCGCv2, and Draco, our method requires more time. While compared with VPCC and QuantDeepSDF which can compress the entire dataset at once, the compression process of our NeCGS is faster. We also want to note that according to the quantitative comparison in Fig. 4 and qualitative comparison in Fig. 5 of the manuscript, the compression performance of our method is significantly better than that of the baseline methods. |Method|Compression Time (h)| |-----|----| |GPCC| 0.625 | |VPCC| 39.34 | |PCGCv2| 1.76 | |Draco| 0.03 | |QuantDeepSDF| 18.91 | |Ours|16.32 |
• Again, we want to emphasize that our NecGS is designed for the offline compression of 3D geometry datasets to save storage space, where the optimization/compression time should not be a key factor. Instead, we should concentrate more on the decompression/decoding speed because the users expect to obtain the 3D models timely when querying the dataset. As shown in Table 3, when the resolution is 128, the inference time of our NeCGS is only 98.95ms, satisfying the real-time requirement.

Comment 5. Line 123, "mplicitly" should be "implicitly".

Response: Thanks. We will correct this typo.

Dear Reviewer qNSe

Thanks for your time and effort in reviewing our manuscript. In our previous response, we addressed your concerns directly and comprehensively. We very much look forward to your further feedback on our responses. Let us discuss.

Best regards,

The authors

It is wonderful to get your further feedback mentioning that our initial responses have effectively tackled your concerns. The authors appreciate your favorable recommendation with the highest confidence. Moving forward, we will tackle the remaining questions you have.

Comment 1. However, I don't feel it is an apple to apple comparison as the geometry complexity of the model in Fig 5 of [5] is much higher than the complexity of models shown in this paper.

Response As shown in our response, we have preliminarily conducted the experiment about [2] and [5] using their released codes on the Thingi10K dataset, when reducing the feature dimension to make the model/feature size to KBs, we cannot extract meshes from them. We will also explore them comprehensively and fairly in the final version.

Comment 2. In addition to this, I also suggest the authors to benchmark the models generalization vs compression time trade off on a larger and complex pool of shapes to get a better picture.

Response Thanks for your valuable comments! We will test the compression performance and generalization of our algorithm on larger and more complex shapes in the final version. Additionally, we will build a benchmark through these data to support the advancement of this field.

Finally, we appreciate the valuable comments and timely feedback from the reviewers.

Comment 1. this paper over claims what it does. in L1-3, it says that they made the first attempt to tackle the problem of compressing 3D geometry sets containing diverse categories. this isn't true. there are at least two papers doing geometry compression of 3D geometry [a], [b].

Response: We strongly disagree with you, due to the following facts.

• After a comprehensive survey, we confirm that all previous methods for 3D geometry compression, including the mentioned [a] and [b] for processing single 3D shapes, have primarily focused on either individual or sequential/dynamic 3D models. Different from previous methods, our NeCGS is designed for the offline compression of 3D geometry sets comprising various unrelated 3D shapes.
• Technically, our NeCGS is significantly different from [a] and [b]. Specifically, [a] employs an individual MLP to regress the implicit field of a single model, using the MLP weights to represent the model. Meanwhile, [b] is designed for mesh simplification and restoration through triangle merge and divide, with the simplified meshes serving as representations of the original meshes. Differently, to well handle each model of a dataset usually containing 3D models with various structures, we propose innovative TSDF-Def volumes to represent 3D models as structured 4D tensors. TSDF-Def can not only represent 3D models with relatively simple structures like TSDF but also well preserve intricate details of 3D models with complex and fine structures. After optimizing each shape into a TSDF-Def volume, we design an auto-decoder structure to regress these tensors, where the embedded features and decoder weights are quantized during the optimization and encoded to bitstreams to represent the whole geometry set.

Comment 2. [a] and [b] are very important references but they are not cited nor discussed. it's not necessary to compare the proposed method with [a] and [b], but at least the authors should acknowledge the existence of these two papers

Response:

• The authors confirm that [a] and [b] are not the most relevant references. But we are fine with citing these two papers in the final version.

• In addition to the response to Comment 1 that has clearly elaborated the differences between ours and [a][b], we further clarify that both [a] and [b] emphasize geometric representation over compression techniques. Besides, there are so many methods for compressing single 3D models, and we have cited well-known methods; however, due to page limitations, it is impossible to cite all of them.

Comment 3. optimization time is too long

Response:

• We must emphasize that our NeCGS is focused on the offline compression of geometry datasets to save storage space, where the optimization/compression/encoding time should not be a key factor. Instead, we should concentrate more on the decompression/decoding speed because the users expect to obtain the 3D models timely when querying the dataset. As shown in Table 3, the decompression process of our NeCGS is extremely fast, allowing for real-time invocation.

• Moreover, there are various potential solutions to accelerate the optimization. At the software level, more efficient convolution (e.g., using multiple 1-D or 2-D convolution to approximate the 3-D convolution or using 3D sparse convolution to replace the traditional 3D convolution) can be used to speed up the process. At the hardware level, optimization programs can be run using multiple GPUs or other more efficient hardware. Like NeRF, the initial algorithms required several days for optimization. Subsequently, improved methods have been introduced to accelerate the optimization process to a few minutes or even seconds.

Comment 4. it is unclear whether the proposed method is reproducible

Response:

• We have included the source code in the submitted supplementary material, as highlighted in the abstract,
• We have provided sufficient implementation details in Sec. 4.1. Based on the information provided, we believe any qualified researcher can replicate the results in our paper.
• Reviewer qNSe acknowledged the reproducibility aspect by stating, 'Reproducibility: All the details to replicate the results are provided along with the code and architecture details in the supplementary material'. And Reviewer 18DA also recognize the reproducibility by stating, 'The inclusion of source code in the supplemental material enhances the reproducibility and transparency of the research'.

Comment 5. typo L43: Matching cubes $\to$ Marching cubes

Response: Thanks for this valuable comment. We will correct the typo.

Dear Reviewer tAuE

Thanks for your time and effort in reviewing our manuscript. In our previous response, we addressed your concerns directly and comprehensively. We very much look forward to your further feedback on our responses. Let us discuss.

Best regards,

The authors

RE: Response to comment 1.

I do not think the authors fully understand the references I pointed out. Both [a] and [b] can do offline compression and were tested on Thingi10K which consists of various unrelated 3D shapes.

We further clarify that both [a] and [b] emphasize geometric representation over compression techniques

I do not see how representations and compression techniques are disentangled in this case. Choosing the right representation is part of the compression technique.

RE: Response to comment 3.

It doesn't really matter whether your method is online or offline. If one wants to apply your method to some new shapes and they don't want to lose accuracy, they'll have to go through the optimization process and this process takes a lot of time. It doesn't really matter if your method can uncompress fast.

Thanks for your quick action.

Comment 1. If you could look at [b], they have one shared network for all 3D models.

Response You still misunderstood [b]. Could you please take the time to read it carefully? As replied in our previous posts, [b] simplifies the meshes one by one, like the compared baseline method PCGCv2. More specifically, [b] trains a shared encoder to simplify the raw meshes one by one, like PCGCv2. [b] and our NeCGS have totally different working mechanisms. Our NeCGS utilizes an auto-decoder framework, where the embedded features and decoder weights are optimized to represent the whole dataset. Besides, our method concentrates on implicit representation, while [b] concentrates on explicit representation, i.e., triangle merge and divide. We believe your confusion is completely caused by your misunderstanding about these methods and lack of knowledge of this topic.

Comment 2. Every reviewer raised different questions and it requires the authors to post such long responses to clarify things.

Response We are wondering whether you have read and fully understood their comments and our responses. We answered all questions directly and comprehensively and provided additional necessary experiments.

Comment 3. Not to mention, in this case, we already have a few back-and-forth discussions but there is still a lot of confusion. I strongly suggest the authors revise the way of presentation.

Response We believe all confusion is due to your lack of relevant foundational knowledge and failure to carefully read the related papers. We believe our explanations have been sufficiently clear, and we strongly recommend you carefully read the paper, supplement the necessary foundational knowledge, and become a more professional reviewer, so that the entire community can progress more effectively.