Thanks for the questions and comments. Please see the following responses.

Q1: Novelty and relevant methods discussion.

Thanks for this question. The major novelty of our approach is to learn the equivariant implicit vector fields for 3D reconstruction. Our novelty lies in the motivation and idea to design the patch-based intrinsic feature learning network. Our framework is inspired by the observation that local patches of 3D shapes repeatedly appear on 3D surfaces if removing the poses, we thus conduct pose normalization, and design the patch feature learning and learnable memory bank for learning the intrinsic patch geometry features. With them, the learned implicit vector fields are equivariant. These designs enable us to achieve state-of-the-art reconstruction results while being robust to the rotations of input points, as shown in the experiments.

Compared with [1], our approach achieves equivariance for the vector field prediction. Compared with [2], our model is equivariant by proposing intrinsic patch geometric feature learning, instead of using vector neurons in [2]. Compared with [3], we tackle different tasks (classification vs. 3D reconstruction), and the model designs are significantly different. We will more clearly discuss the relation and novel contributions compared with these related works in the paper.

Q2: The Robustness of PCA-based alignment and comparison with other alternatives, such as Vector-Neurons[4] and SE(3)-transformers [5].

Thanks for this good question. In our approach, we compute the PCA over local patches of local points. As suggested, we conduct experiments to test the robustness of PCA-based alignment, and compare with other alternatives using Vector-Neurons[4] and SE(3)-transformers [5].

(1) To test the robustness of PCA, when computing the PCA over patches, we randomly perturbed the patch point coordinates by Gaussian noises with $\sigma=0.001$ (the average distance from knn points to patch center point is about 0.004), resulting in perturbed PCA matrices. The results on the ABC dataset in Table R4-1 show the robustness of PEIF to PCA perturbations.

Table R4-1. Comparison of results for random perturbations of rotation matrices.

(2) Using the same GPU, we substitute the layers of our network with the Vector-Neurons[4]-based layers, and set hyper-parameters fitting the GPU memory. The results are in Table R4-2. Table R4-3 provides the hyper-parameters and resource consumption of our model and Vector-Neurons[4]-based implementation.

Table R4-2. The results of PEIF with Vector-Neurons [4] equivariant layers.

Table R4-3. The hyper-parameter comparison of Vector-Neurons [4] and Pose-normalization based PEIF on ABC dataset with a single NVIDIA 4090 GPU.

(3) We also attempt to use the SE(3)-transformers [5] to replace the patch feature extraction (SRM, PFEM) in our model. Using the same GPU, even by setting the batch size to 1 and the feature dimension of 64, we experienced out-of-memory issues when training the SE(3)-transformer based implementation.

As a summary, the PCA-based patch pose normalization enables us to design a light-weight network achieving equivariant implicit function learning, and achieves sota results shown in experiments.

Q3: Further comparison with E-GraphONet [2].

For a fair comparison, we tried to use PEIF to predict the occupancy field (OF) by only changing its loss to regress OF. In such a setting, we use the same training/test dataset as E-GraphONet. The reconstruction results are reported in Table R4-4. We also trained E-GraphONet to predict the vector field, we tried our best but the learned model cannot reasonably reconstruct object surfaces. The integration of training based on both occupancy and vector field prediction deserves us to try in the future.

Table R4-4. The reconstruction results of PEIF predicting occupancy field on ABC dataset.

Q4: Comparison of PSR [6] and similar follow-up works on the ABC dataset.

As suggested, we compare with PSR [6], OccNet [7], and SAP [8] on the ABC dataset in Table R4-5. Our PEIF achieves the best results.

Table R4-5. The reconstruction results on ABC dataset.

Q5: Typos.

Thanks and we will fix these typos.

[1] Yang et al. Neural vector fields: Implicit representation by explicit learning. CVPR. 2023.
[2] Chen et al. 3D Equivariant Graph Implicit Functions. ECCV. 2022.
[3] Zhao et al. Rotation invariant point cloud classification: where local geometry meets global topology. Pattern Recognition. 2021.
[4] Deng et al. Vector neurons: A general framework for so (3)-equivariant networks. ICCV. 2021.
[5] Fuchs et al. SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks, NeurIPS. 2020.
[6] Kazhdan et al. Poisson surface reconstruction. Proceedings of the fourth Eurographics symposium on Geometry processing. 2006.
[7] Mescheder et al. Occupancy networks: Learning 3d reconstruction in function space. CVPR. 2019.
[8] Peng et al. Shape as points: A differentiable poisson solver. NeurIPS. 2021.

Thank you for the positive feedback on our rebuttal, and the additional comments/questions. We further respond to these questions as follows.

(1) Vector neuron-based method constructs equivariant neural layers by extending neurons from 1D scalars to 3D vectors, thereby ensuring equivariance when implementing SO(3) actions in vector-based feature space. This higher-dimensional vector-based representation requires higher momery cost, more matrix operations and vector transformations to maintain geometric properties and rotation invariance, and consumes more computational cost.

As suggested by the reviewer, we will analyze and discuss the computational cost considering the other computationally expensive equivariant alternatives, and clarify the motivation on using PCA for patch pose normalization, as a simple yet effective strategy for learning equivariant implicit function, while yielding good experimental results.

(2) Thanks for the question on the comparison of expressiveness of PCA-alignment and VN-based representation. We think that the current experiments can hardly disentangle the effects of PCA/VN from the network design and conclude on the expressiveness comparison between them. In our model, the design of PCA-normalization at patch level is tied with our specific network design. By removing the poses of local patches using PCA-alignment, we propose the modules of "patch feature extraction module" , "intrinsic patch geometry extractor" for learning the patch-level intrinsic geometric features, as well as the "spatial relation module" encoding the off-set-based features between query point coordinate to its neighboring patch (kNN). These designs are integrated with the PCA-alignment for achieving the off-set vector field estimation. The comparative analysis between the PCA-alignment and VN is an interesting topic, and we are interested to design experiments that can disentangle the effects of PCA/VN from the specific network designs to compare them. Due to the limited time, we will consider this in the future work.

Thanks for the valuable comments and suggestions. Please see below for the responses.

Q1: A heuristic baseline.

We aim to learn the equivariant implicit function that outputs the vector of each query point to its nearest point on the unknown continuous 3D surface. Since only discrete points on the surface are observed, it is challenging to infer the vector of the query point to the continuous 3D surface based on these discrete points. Instead of using local knn points to regress a local surface function (e.g., polynomial functions) for each query point, our neural network-based approach infers the vector of the query point to the surface by learning the intrinsic geometry feature of the local patch to estimate the vector of each query point. We also design and take advantage of the rotation-invariant patch features for achieving equivariant implicit function learning. This design enables us to achieve sota performance for 3D reconstruction. In Q2, we also evaluate the performance of the degradations of input point clouds.

Q2: The performance of PEIF on sparse/noisy/partial data.

As suggested, we evaluate the performance of our PEIF on different data degradations (sparse, noisy, and partial) on the ABC dataset.

In the following experiments, the test input point clouds with different degradations, and the results are reported in Tables R3-1 to R3-3. In Tables 1-3 of the uploaded PDF file in “general response”, we also report the results when both the training and test point clouds are degraded.

(1) Sparse point cloud. In experiments of the original submission, all the compared methods use the same number of input points (10k) for each shape in testing, as NVF. We randomly select a subset of input points as input, and the results are in Table R3-1.

Table R3-1. The reconstruction results of sparse data on the ABC dataset.

(2) Noisy point cloud. We plugged Gaussian noise with standard deviation ($\sigma$) as 0.005 and 0.01 to the input points.

Table R3-2. The reconstruction results from noisy input on the ABC dataset.

(3) Partial point cloud. We remove a fraction (with ratio $p$) of the input points to form a partial point cloud. Specifically, we use the farthest point sampling to select a set of center points and remove their K-NN points to ensure the sampling fraction.

Table R3-3. The reconstruction results from partial points on the ABC dataset.

We thank the reviewer for the positive comment that our approach is well-motivated and novel. Please see below for responses.

Q1: The experimental settings of "w/o rotation" and "w/ rotation" in Table 4.

In Table 4, "w/ rotation" and "w/o rotation" represent that the testing input point cloud is with and without arbitrary rotation respectively. This will be clarified in paper.

Q2: Missing citations of related works in 2D pixel-level.

Thanks for this comment. RELF [1] uses group equivariant CNNs to extract discriminative rotation-invariant descriptors for 2D images. Differently, our framework is designed for 3D reconstruction, and we design equivariant implicit function learning for 3D reconstruction. We will include the reference of equivariant network in 2D and discussion of RELF [1] in our paper.

Q3: Computational cost.

We report the computational cost, including the training time per epoch, training memory, testing time per 3D shape, and testing memory cost on the ABC dataset, in Table R2-1. Methods of GeoUDF and GridFormer include two stages of upsampling/reconstruction and reconstruction/refinement. We report the computation cost of them in each table cell with two values (denoted as · + ·), respectively representing the costs for each stage. We will include these details in the appendix.

Table R2-1. The comparison of computational cost.

Q4: Further research direction.

It is a good suggestion to extend our approach to few-shot training scenarios. Along this direction, our patch-based pose invariant representation can be taken as a foundation network for pre-training, followed by fine-tuning on few-shot examples. In the pre-training step, we may learn a general representation of intrinsic 3D patch features, and the fine-tuning may adapt these representations to the given few-shot examples. We will include this direction as a future work in the conclusion section.

[1] Lee et al. Learning rotation-equivariant features for visual correspondence. CVPR. 2023.

Thank you for the valuable questions and the positive feedback. As suggested, we will discuss the related work of equivariant/invariant mapping in 2D image matching [1] in the final revision.

Thanks for these comments. We address the concerns and questions as follows.

Q1: The visualization of the learned memory bank.

We provided two approaches for visualizations of the learned memory bank. Please refer to Figure 1 in the attached PDF file uploaded in the top “general response”.

(1) We visualize the set of point patches with the highest weights to the corresponding element of the memory bank (the weights are computed by Eqn. (12)). These patches are highlighted by colors in these examples. It shows that the patches with high weights to each element of memory have similar geometry structures.

(2) We further visualize (by t-SNE) the features of point patches with the highest weights (Eqn. (12)) to different elements of the learned memory, rendered by different colors. It shows that the patches with high weights assigned to different elements of learned memory have clustered features in the feature space.

We will include these visualizations in the appendix of our paper.

Q2: What value of $K$ is used in the MGN experiments?

$K$ is set to 54 in testing on the MNG dataset, using the trained model with $K=54$ on Synthetic Rooms dataset. As shown in Table R1-1, when changing $K$ to 48 and 32 on the Synthetic Rooms in training, the test results using the corresponding $K$ on MGN are stable. We also presented the ablation studies on $K$ in Table 5 of the manuscript.

Table R1-1. The impact of $K$ when training on Synthetic Rooms and testing on MGN.

Q3: Minor comments on typos.

Thanks and we will correct them.

Q4: A detailed analysis of the inference time.

In Table R1-2, we report the time consumption of each operator in PEIF to process 10,000 query points. Specifically, the operations include SVD (Singular Value Decomposition), PE (Point-wise Feature Extraction), SRM (Spatial Relation Module), PFEM (Patch Feature Extraction Module), IPGE (Intrinsic Patch Geometry Extractor) and Others (other Conv layers).

Table R1-2. The time to process 10,000 points on the ABC dataset using one NVIDIA 4090 GPU.

We will add them in the main body or appendix of our paper.

Q5: Comparison with GeoUDF and necessity of equivariance.

As shown in Tables 1, 2 of the manuscript and Tables R3-1 to R3-3 in response to Reviewer H6Ua, our results are better than GeoUF, especially for degraded data. Moreover, Table R2-1 in response to Reviewer 6vbf shows that our time/memory cost is lower than GeoUDF.

(1) Learning the equivariant implicit representation ensures that the reconstructed surfaces are robust to the rotation of the input points. Figure 2 of the uploaded PDF in “general response” shows that GeoUDF generated 3D surfaces with some noise artifacts after rotation. While our results are more smooth and stable to rotations.

(2) The equivariance/invariance is important to our model's performance. We learn the equivariant implicit vector field based on patch intrinsic geometric features by removing patch poses. It is inspired by the observation that local patches are repetitively appearing on 3D shapes if removing patch poses. This novel idea is essential for achieving improved reconstruction accuracy, while robust to rotations. As shown in Table 5, if removing the equivariance design (i.e., w/o pose normalization), the 3D reconstruction accuracy apparently decreases.

Q6: The comparison of cross-domain generalization ability with E-GraphONet.

As suggested, we compare with E-GraphONet in the cross-domain experiment as reported in Table R1-3. Our PEIF achieves better quantitative results on the MGN dataset. These results will be included in Table 3 of our paper.

Table R1-3. The cross-domain evaluation on the real MGN dataset, where the model is pre-trained on the Synthetic Room dataset.

Q7: Further clarification on the robustness of the competed algorithms to rotation changes.

Thanks for the question. We have further evaluated the rotation robustness of the methods in Table 4 to different rotation angles, which is reported in Table R1-4. Figures 2, 3 of the uploaded PDF in "general response" demonstrate that NVF and GeoUDF exhibit noise and holes after rotation, while our results are more smooth and robust to rotations. In Q5, we also discussed the importance of equivalence/invariance of vector field/patch features to our model for achieving good performance.

Table R1-4. The results for different rotation angles (0/90/180/270) on the ABC dataset.

Q8: An ablation study of the three modules.

We conduct an ablation study of the three modules (e.g., SRM, PFEM, IPGE) in our PEIF on the ABC dataset. Table R1-5 reports the quantitative measures of our PEIF without these modules. The results show that the intrinsic patch geometry extractor (IPGE) contributes more to the performance of PEIF.

Table R1-5. Ablation study of the three main modules proposed in PEIF.

Dear reviewer dfBj,

Thanks for your inspiring suggestions and questions. We have carefully considered your questions/concerns, and responded in the following aspects. (1) We have provided the visualization of the learned memory bank, showing that the elements of the memory bank represent patch-level 3D geometric patterns. (2) We have presented more details on the inference time, the setting of K, etc. (3) We have provided additional experiments on the justification of the necessity of equivariance and robustness to the rotations. (4) We have conducted ablation studies on the key modules and the cross-domain comparison with E-GraphONet. Due to the limited remaining time for authors-reviewer discussion, we are expecting to have further discussion with you if there are any additional concerns.

Kind regards,

All authors

Thank you for the positive feedback. We will incorporate the corresponding revisions into our paper.