MultiPull: Detailing Signed Distance Functions by Pulling Multi-Level Queries at Multi-Step

Thank you for your acknowledgements in our method and results. In the following, we respond to the comments with respect to the weaknesses and questions in turn. All experiments presented in the rebuttal are based on the FAMOUS dataset.

(1) Further comparison with related works.

The field of surface reconstruction from point clouds is quite vast, so the authors missed several works

RESPONSE: We follow the reviewer's suggestion to make a further comparison of the related works. Due to the time limit, we just select overfitting based method Neural-Singular-Hessian [1] for visual comparison, and we will add additional comparison results of other work in our version. As shown in Fig. C in the PDF, our method still has competitive performance compared to Neural-Singular, and we perform better in generating local details.

(2) Concern about frequency features.

It is unclear to me why the SDF evaluated at $Q_1$ uses a feature $y_1$ as input that was computed from a query position $Q_0$, rather than a feature computed at query position $Q_1$. An ablation would also be interesting: how would the method perform when re-evaluating $h_0$, ..., $h_i$ at $Q_i$ instead of $Q_0$ when computing $y_i$ for f($Q_i$, $y_i$)

RESPONSE: All frequency features y in the FFT are obtained from the query point $Q_0$. we transform the query point $Q_0$ into frequency features as conditions to guide the movement of $Q_0$, $Q_1$, and $Q_2$. The deeper the features in the FFT, the more comprehensive the information contained. We hope to use this movement strategy to enable MSP to learn the global information of the input from low-frequency features, and then optimize local details through high-frequency features. In summary, the frequency features of FFT come from the same set of query points. If we re-evaluate the $Q_i$ of each stage into FFT, it will disrupt the consistency of the features. Therefore, FFT cannot simultaneously accept multiple point clouds for frequency feature encoding.

(3) Details of $\mathcal{L}_{recon}$.

In Eq. 6, the design choice of only scaling $D_1$ and $D_2$ with the error-based factor, but not $D_3$ needs to be motivated. How would Eq. 6 look for a general $N_L$., for example?

(4) Design choice for $\mathcal{L}_{grad}$.

In Eq. 7, the design choice for using the minimum over the three levels.

RESPONSE: In order to maintain consistency in the moving direction of query points at different steps during continuous offset, we naturally choose gradient similarity $L_{grad}$ as a constraint term. Furthermore, we force the network to focus on the step where the gradient direction changes the most. We conduct experiments according to reviewer uTM5 to demonstrate the validity in Tab. A.

RESPONSE: We force network to focus more on outliers and points close to the surface to accelerate network convergence in $L_{recon}$. Therefore, we make the error-based factors of $D_3$ higher than and $D_1$ and $D_2$. We define a set $D$={$D_{1}, D_{2}, D_{3}$} and its corresponding weights $w$={$\alpha, (1- \alpha )^{\gamma}, 1$}, then $L_{recon}$ can be expressed as: $L_{recon}$ = $\sum^{3}_{i=1}w_i D_i$.

(5) Inference time comparison.

An inference time comparison between methods should be included.

RESPONSE: In Tab. G, we further evaluate inference time with NP and PCP. The results show that the local based method (PCP) has the longer inference time than the global based method (NP, Multiplaull), with NP having the fastest inference time.

(6) Explaination about details.

The exposition is hard to understand in some parts and could be improved.

RESPONSE: Thank you for the reviewer's suggestions for writing revisions and reminders for adding details. Here we will present the issues that require further discussion, and we will refine their writing suggestions and revision suggestions in the next version.

Differences of GridPull.

Our method uses pulling based loss like GP [2]. However, our optimization strategy is different from GP. GP uses grid points to perform trilinear interpolation on point inputs to predict the SDFs, which does not require any neural network for optimization. Instead, We need multi-scale frequency features as conditions to move the query points toward the underlying surface of the point cloud in multiple steps. This process uses LSNN network to learn SDFs and gradients at different steps. As shown in the visual comparison in the main text, optimization based methods can learn smoother surfaces.

Architecture of LSNN.

LSNN consists of 8 linear layers, with seven layers having a dimension of 512, and the last layer having a dimension of 1. We use LSNN to predict the signed distance value $f (Q_i, y_i)$ of the query point $Q_i$ under the frequency feature $y_i$ after the $i-th$ step of movement. We will add relevant content in the subsequent appendix. We apologize for any inconvenience caused to your reading.

Construction of query points.

We construct query points based on the sampling method of GridPull [2]. Due to space limitations, reviewer can refer to reviewer NaoM's question (6) for a detailed introduction.

Scope of use of $\mathcal{L}_{surf}$.

We only use $\mathcal{L}_{surf}$ at $f(Q_3)$ to make the network more confident in predicting the zero-level-set on the point cloud surface.

[1] Wang Z, Zhang Y, Xu R, et al. Neural-Singular-Hessian: Implicit Neural Representation of Unoriented Point Clouds by Enforcing Singular Hessian[J]. ACM Transactions on Graphics (TOG), 2023, 42(6): 1-14.

[2] Chen C, Liu Y S, Han Z. Gridpull: Towards scalability in learning implicit representations from 3d point clouds[C]//Proceedings of the ieee/cvf international conference on computer vision. 2023: 18322-18334.

Thank you for your acknowledgements in our method and results. In the following, we respond to the comments with respect to the weaknesses and questions in turn. All experiments presented in the rebuttal are based on the FAMOUS dataset.

(1) Motivation of FFT.

The rationale for using frequency features to represent query points needs further clarification.

RESPONSE: Previous works fail to accurately represent the local geometry of objects by learning global features. While adopting local methods can alleviate this issue, they typically suffer from high parameter counts and extended training times. Inspired by LOD models, we transform complex features into a more intuitive and interpretable frequency domain representation for neural networks. By utilizing simple frequency domain feature transformations and linear layers (Equation (2) and (3) in the main text ), we can effectively represent features at different scales, it also aids in predicting implicit fields from coarse to fine, where high frequency features help produce more sensitive adjust in signed distance field, leading to high frequency geometry details.

(2) Details about LSNN.

While the use of multiple levels of query points is understandable, the choice of LSNN at each level and how query points are used in this model require more detailed explanation.

RESPONSE: LSNN is a neural network consisting of 8 fully connected layers with ReLU activation functions. The first 7 layers have a dimension of 512, while the last layer has a dimension of 1. We use it to predict the SDFs and gradient of query points at different steps. The FFT module encodes the query point $Q_0$ into frequency features $y$ at different scales. We select three of these frequency features as conditions for the multi-step offset of the MSP module's query point to predict the implicit field $f$ after the offset. We denote the $i$-th step query points and frequency feature as $Q_i$ and $y_i$, respectively. We concatenate $Q_i$ and $y_i$ and use the LSNN network with shared parameters to predict the signed distance $f(Q_i, y_i)$ and the gradient $\nabla f(Q_i, y_i)$. In this way, we finally obtain the different levels of query points $Q_1$, $Q_2$, and $Q_3$.

(3) Analysis of Results.

From the visualization, the proposed method seems to produce results with fewer holes and smoother surfaces. Is this due to the model learning better features, or is there a specific loss design different from prior work?

RESPONSE: We analyze the effectiveness of different modules from two aspects: (1) We conduct ablation experiments on modules other than MSP in Tab. E. We first remove the FFT module and use linear layers to learn query point features for reconstruction, resulting in a significant increase in CD error, which proves the effectiveness of the FFT module. Next, we gradually remove different $L_{sim}$and $\mathcal{L}_ {surf}$to explore the importance of these constraint terms. After that, we obtain slightly worse results after removing two losses. (2) We further explore the impact of MSP on accuracy. As shown in Tab. B, the CD error increases significantly when the steps are reduced, and we get the worst result when $Step=1$. Overall, the offset steps and FFT module have the greatest impact for our method.

(4) Feature Comparison.

Could you compare the MSP using the same features from prior work to demonstrate its performance even without the proposed feature? This would help to convincingly show its effectiveness.

RESPONSE: We followed the reviewer's suggestion to combine MSP with the linear layers and the traditional feature learning encoder PointMLP [1] to explore the effectiveness of MSP. We present the results of combining different feature learning networks with MSP methods in Tab. F. We denote the single moving operation in MSP as Pull and use multiple feature learning networks as baselines. Due to the lack of an FFT module, the same features are used for multiple offsets in linear+MSP and PointMLP + MSP. As shown in Tab. F, the combination of different feature extraction networks and MSP achieved better results in terms of both CD and NC metrics. We further demonstrate the effectiveness of MSP in Fig. A and Fig. B in the PDF. PointMLP / linear+MSP can generate finer local details compared to PointMLP / linear + Pull.

(5) Typo correction.

S is used for both raw point cloud in L91 and SDF surface on L99. D is used for both feature dimensions and distance. Minor: The number reported on the last row of Table 7 appears to be wrong

RESPONSE: Thank you for the typo errors raised by the reviewer. We will correct these errors in future versions. We apologize for any inconvenience caused during your reading.

[1] Ma X, Qin C, You H, et al. Rethinking network design and local geometry in point cloud: A simple residual MLP framework[J]. arXiv preprint arXiv:2202.07123, 2022.

Thank you for your acknowledgements in our method and writing. In the following, we respond to the comments with respect to the weaknesses and questions in turn. All experiments presented in the rebuttal are based on the FAMOUS dataset.

(1) Concern about frequency feature consistency.

There seems to be no guarantee that the scale of the displacement of the query point at each frequency varies with the frequency of interest. Is there some possible constraint on the magnitude of the slope that would be consistent with the frequency of interest?

RESPONSE: Yes, your understanding is correct. There is no guarantee for that. But one thing we observed during experiments is that the movement is getting smaller and smaller for the last several pulling since the pulled queries are approaching to the zero-level set. Thus, we use high frequency features as conditions to make the network more sensitive, so that we can capturer high frequency geometry details. We extract different levels of frequency features for the query points in the FFT module and guide multi-step movements in the MSP. As the reviewer mentioned, we separate the frequency features at different scales, so our constraint on frequency feature consistency mainly lies in the direction consistency during multi-step movements. Throughout the continuous displacement process, we use $\mathcal{L}_{grad}$ to constrain the movement direction of the query points at different steps, ensuring consistency across different frequency features.

(2) Design of $\mathcal{L}_{grad}$.

For the purpose of consistency, would it not be appropriate to use the average or correlation rather than argmin to select one of $Q_1$, $Q_2$, or $Q_3$?

RESPONSE: Ideally, the movement directions of query points at different steps ($Q_1$, $Q_2$, and $Q_3$) should keep consistent. We ensure this consistency by calculating the gradient similarity loss $L_{grad}$. To prevent significant deviations in the moving direction during training, we use argmin as a constraint, making the network more sensitive to changes in gradient direction. To further illustrate this, we use argmin as the baseline and compare it with using the average in $\mathcal{L}_{grad}$. As shown in Tab. A, when using the average to calculate the similarity of query points at different steps, the CD error slightly increases. Conversely, argmin achieves a similar level of similarity but leads to better performance in terms of the CD metrics.

(3) Suggestions of Introduction.

In the introduction, it would be desirable to specify whether the proposed method is a test-only optimization. Figure 3 in [1] clearly shows the position of the proposed method in relation to related research. My understanding is that the paper plots to the same point as NGLOD, and it is preferable to explicitly state that only test time is considered.

RESPONSE: As the reviewer mentioned, our method can learn signed distance functions without introducing extra priors, which is an overfitting based method. In our revision, we will further clarify the differences between the various types of deep learning networks mentioned in the introduction. Additionally, we will provide clear classification explanations based on the references you provided.

(4) Default setting of Step.

After considering both performance metrics and time efficiency, we have set Step=3 by default.", can the authors discuss this trade-off?

RESPONSE: We set Step to 3 by default for two reasons: (1) MSP can bring partial accuracy improvement when learning more steps, but more network parameters are also needed. We present comparisons under different steps in Tab. B. (2) Although deeper frequency features can represent more comprehensive information, combining frequency features from different layers can achieve better results. We denote the combination of the frequency features as ${L_1, ..., L_n}$. We conduct the experiment according to the default settings, and the results are shown in Tab. C.

(5) Related works.

"However, inferring implicit functions without 3D supervision requires a very long convergence process, which limits the performance of unsupervised methods in large-scale point cloud data." Can the author provide a specific reference in the paper?

RESPONSE: Learning accurate implicit functions with an unsupervised strategy is challenging. Here, we provide some related methods as references: CAP-UDF [2], NP[3], and PCP [4].

(6) Construction of query points.

"To this end, we constrain query points to be as close to their nearest neighbor on S." I understood sampling from the nearest neighbor, but did not understand how Q is constructed.

RESPONSE: We construct query points based on the sampling methods of NP. For the input point cloud $s_i \in S$, we sample 80 query points around $s_i$ based on the Gaussian distribution $\mathcal{N}(s_i, \sigma^2)$. We use $\sigma^2$ to control the distance of the query points, we follow NP to set $\sigma^2$ as the square distance between $s_i$ and its 50-th nearest neighbor. We set the default distance as $0.5 \sigma^2$.

[1] Francis Williams, Zan Gojcic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, Or Litany. Neural Fields As Learnable Kernels for 3D Reconstruction. CVPR2022.

[2] Zhou J, Ma B, Li S, et al. CAP-UDF: Learning Unsigned Distance Functions Progressively from Raw Point Clouds with Consistency-Aware Field Optimization[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.

[3] Ma B, Han Z, Liu Y S, et al. Neural-pull: Learning signed distance functions from point clouds by learning to pull space onto surfaces[J]. International Conference on Machine Learning, 2021.

[4] Ma B, Liu Y S, Zwicker M, et al. Surface reconstruction from point clouds by learning predictive context priors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 6326-6337.

Thank you for your acknowledgements in our method. In the following, we respond to the comments with respect to the weaknesses and questions. All experiments presented in the rebuttal are based on the FAMOUS dataset.

(1) Novelty.

In general, the proposed techniques are just an incremental combination of the pulling-based SDF learning approach [18] (Neural-Pull) and the multiplicative fourier learning approach [14] (MFLOD), although with some marginal modifications in order to combine them. and particular sentence is from the paper of MFLOD.

Although we use feature extraction in MFLOD and pulling operation in NeuralPull as basis, simply combining these two operations does not work well due to unstable optimization or poor performance. Hence, our key contributions include the initialization scheme and multi-step pulling (MSP) modules with multiple constraints, rather than a simple combination. First, our initialization scheme can produce more discriminative frequency features and make each Fourier layer more stable and robust under different frequencies, adapting to more complex geometries, while avoiding gradient vanishing. Second, our MSP module with multiple constraints can infer more accurate SDFs and more consistent gradients using frequency based features as conditions. Whereas NeuralPull infers the SDFs by one-step pulling, it is difficult to directly and accurately reach the target in one step for queries, espcially for the queries that are far from the surface. In addition, NeuralPull does not consider gradient consistency on different level sets, which drastically decreases the accuracy of SDFs. In contrast, MultiPull infers the SDFs by multi-step pulling, the distance-aware constraint pays more attention to queries far from the surface, together with the help of the surface constraint, allowing us to pull any queries to correct locations on the zero-level set. We will make this more clear in our revision.

(2) Missing Citation.

We were supposed to add a citation for the sentence that we get from MFLOD, but we forgot to do that. We will add a citation to this sentence in our revision.

Dear reviewers,

As the reviewer-author discussion period is about to end, we are looking forward to your feedback on our rebuttal. Please let us know if our responses address your concerns. We would be glad to make any further explanation and clarification.

Thanks,

The authors