Accurate Differential Operators for Neural Fields

This paper introduces a post-processing method to improve the estimation of first- and second-order derivatives on learned hybrid neural fields (e.g., Müller et al., 2022a), a.k.a. implicit neural representations (INRs). By applying the local polynomial regression, the estimates of these derivatives are smoothed w.r.t the size of the local sampling domain and the number of samples in the local domain. An auxiliary approach is also proposed to fine-tune hybrid neural fields by biassing the estimates via autograd toward the previously proposed smoothed operator.

This paper addresses a critical issue in neural fields, namely, the gradient problem, which holds immense significance in computer vision (e.g., 3D reconstruction in NeRF; consider the neuralAngelo paper), geometry processing, and solving PDEs. The authors propose a method to accurately compute differential operators, including gradients, Hessians, and Laplacians, for hybrid grid-based neural fields. This approach effectively mitigates high-frequency noise issues associated with previous hybrid neural field techniques, particularly on high-resolution grids, by locally fitting a low-degree polynomial and computing derivative operators on the fitted polynomial function. The author's approach outperforms both auto-differentiation-based and finite difference-based methods.

In this paper, the authors address the important problem of inaccurate derivatives computed via automatic differentiation (autodiff) from hybrid neural fields. The authors identify high-frequency noise amplification during derivative computation as the core issue. They propose two solutions - a polynomial fitting operator to smooth the field before differentiation, and a method to fine-tune the field to align autodiff derivatives with more accurate estimates. Experiments demonstrate 4x and 20x reductions in gradient and curvature errors respectively over autodiff.

In this paper the authors tackle issues that arise with hybrid (grid-based) neural fields in the computation of differential operators. They emphasize that automatic differentiations leads to inaccurate and noisy derivatives with a trained hybrid neural field $F$. Then, they introduce a local-polynomial to approximate the SDF field in the neighborhood of a query point, in order to take the differential operators of the local polynomial instead of that of $F$. This approach can be useful at test time, but requires computation at each iteration. To tackle this issue, the authors propose a fine-tuning that is supervised with smoothed gradients while preserving the represented signal. They perform some experiments to validate the effectiveness of the local-polynomial and of the fine-tuning, and then show that this increased accuracy in derivative computations has a positive impact on several downstream tasks (rendering, collision simulations, 2d advection).

We thank all the reviewers for their insightful feedback! We have addressed common concerns in this general response. Major changes in the paper are colored red for convenience. We would love to hear your thoughts about our rebuttal, including whether it sufficiently addresses your concerns and questions. If you believe that our rebuttal is satisfactory, it would be great if you could consider increasing your score. Any feedback is welcome and greatly appreciated!

Contributions

We first reiterate our contributions. Our contributions are three-fold:

1. We identify artifacts in the differential operators (e.g., gradients, curvatures) computed from hybrid neural fields such as Instant NGP [A]. We observe that these artifacts arise because even when the neural field approximates the true underlying field well, it has high-frequency noise that is exacerbated by derivative operators.
2. We propose a local sampling and polynomial fitting approach to obtain more accurate differential operators from pre-trained hybrid neural fields in a post hoc manner.
3. To allow downstream pipelines direct access to accurate derivatives through autodiff, we further propose a fine-tuning approach that uses our post hoc operator to supervise autodiff gradients of the neural field, while preserving the original field.

Experiments on other neural field architectures (SIREN, etc.)

Non-hybrid architectures: As we mentioned above, our approach can address high-frequency noise in neural fields, leading to more accurate derivatives. While our approach itself is not tied to any specific network architecture, we found that high-frequency noise is more prevalent in hybrid architectures. In our preliminary experiments, we found that while non-hybrid architectures, specifically SIREN, are also prone to inaccurate derivatives, the reason for these inaccuracies seems to be different from the high-frequency noise that we observe for Instant NGP. High-frequency noise is of a lower magnitude in SIREN. Instead, low-frequency errors seem to be more abundant. Due to this difference, our method is not well-suited for obtaining accurate derivatives from SIREN. Tackling this issue for SIREN would require altogether different approaches which we hope to address in future work. We have made this clearer in the main text (Section 3) and have also added a note about our preliminary experiments with SIREN in Appendix F.

Hybrid architectures: For testing hybrid architectures, we selected Instant NGP to use a standard and well-accepted architecture. However, we have now added experiments with other hybrid architectures, including “dense” grid architectures, i.e., without the hash data structure utilized by Instant NGP, as well as in another hybrid neural field, Tri-planes [B]. We find similar artifacts and demonstrate that our approach can help alleviate them. (see Appendix C, under “Analysis on other hybrid neural fields”).

Computational Cost

Reviewers DXqV, 6pQ5 had questions about the computational cost of our method.

We have provided a run-time analysis of our post hoc operators in the supplementary (Appendix C), which shows that our post hoc operators perform competitively with the baselines. To answer reviewer 6pQ5’s question, we use Pytorch’s internal least squares solver which internally chooses between LU or QR factorization for computation.

For our fine-tuning approach, in each training step, our post hoc operator computes accurate gradients for the sampled points and then uses them for training the autodiff gradients. The overall training takes 4k steps or ~2 hours. However, we already get ~90% of the reported performance in ~1k steps or ~30 minutes. These times are reported for the standard Instant NGP architecture. Using smaller grids can further improve this time.

Reviewer DXqV further questioned if our approach requires training our model twice. Note that the pre-trained hybrid neural field can perfectly represent the zeroth-order signal (eg, the shape SDF or the initial value function in PDE solving). We only require an additional fine-tuning step to ensure accurate AD derivatives. This is because our approach is self-supervised using the accurate zeroth-order signal learned by the signal. This is a one-time measure, and subsequent AD queries do not require training again. Lastly, if computational resources are a concern, another option can be to use our post hoc operators which do not require any further training of the pre-trained model.

[A] Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. Müller et al. SIGGRAPH 2022.

[B] Efficient Geometry-aware 3D Generative Adversarial Networks. Chan et al. CVPR 2022.

As one of the reviewers advocating for this paper, I'd like to respectfully share my perspective on why I believe this paper should be considered for acceptance.

The hybrid neural field, particularly in the instantNGP style, holds a significant position within the realm of neural fields. Making enhancements within this specific class of neural fields is already very impactful and should be acknowledged. I personally think that a paper need not address all categories of neural fields to merit acceptance. I appreciate the authors' adjustment of the title to emphasize their focus on hybrid neural fields, which has led to a clearer delineation of the paper's scope.

I acknowledge the valuable contributions of the additional baselines highlighted by Reviewer DXqV, which definitely have the potential to address gradient-related challenges in hybrid neural fields. However, to the best of my knowledge, with the exception of the built-in FD, these baselines are not explicitly tailored for hybrid neural fields. Therefore, it may be somewhat unreasonable to expect the authors to provide direct comparisons with them. That said, I concur that comparing with the FD baseline is a relevant choice, given its established usage in the hybrid neural field literature, as exemplified in works such as "Neuralangelo."

The concern raised by Reviewer 6pQ5 regarding oversmoothing in PDE solvers is also valid. However, I think it's important to recognize that similar smoothing issues can manifest in classic numerical methods. For instance, in finite element methods (FEM), the choice of basis functions (whether first order or second order) can result in gradients with varying degrees of smoothness. Consequently, the smoothing issue alone should not be considered a decisive obstacle to numerically solving PDEs. On the contrary, addressing the issue of excessive noise in the original hybrid neural field is definitely a significant concern since it will definitely prevent PDE solvers from running.

In conclusion, I believe the reviewers have brought up very useful and insightful concerns. The authors should certainly consider incorporating these points into their paper, including both in discussions and conducting additional experiments where appropriate. Nevertheless, in my humble opinion, the paper has already effectively defined the scope of the problem it addresses and has provided a valid solution, including a proper comparison with relevant baselines.

• Claims and findings:

This paper tackles an important problem in neural fields (eg. high-frequency noise) which has significance in computer vision (e.g., 3D reconstruction), geometry processing, and solving PDEs. The authors propose a method to accurately compute differential operators, including gradients, Hessians, and Laplacians, for hybrid grid-based neural fields. This approach effectively mitigates high-frequency noise issues associated with previous hybrid neural field techniques, particularly on high-resolution grids, by locally fitting a low-degree polynomial and computing derivative operators on the fitted polynomial function. The author's approach outperforms both auto-differentiation-based and finite difference-based methods.

• Strengths:

Reviewers have pointed out that the angle of the paper is original and the local-polynomial fitting sounds like a simple yet effective idea to smooth out the derivatives of a neural field. In addition, reviewers have also highlighted that the paper offers a range of practical downstream applications and while it may not yield surprising results for most proposed tasks, it does exhibit improvements compared to pre-trained hybrid neural fields.

• Weaknesses:

Reviewers have noted that the claim regarding motivation from empirical observations might be somewhat overstated, particularly when applying it to all types of neural fields as opposed to only hybrid neural fields. In addition, reviewers have also pointed out that the proposed method seems to be completely based on a pre-trained network. Which means, the model is essentially being trained twice.

• Missing in submission: N/A

Reject