From Transparent to Opaque: Rethinking Neural Implicit Surfaces with $\alpha$-NeuS

We thank the reviewer for the comments.

Q: Overview of NeuS

A: We appreciate your feedback and will add the overview of NeuS in our revision. We prepare an initial overview as follows:

NeuS is a surface reconstruction method based on Neural Radiance Field (NeRF) that uses volume rendering technique, which is differentiable, to render the image and optimize the scene representation. In the realm of volume rendering, the color of individual pixels is determined by the color of individual rays that shoots from the camera origin $\mathbf{o}$ and passes through the corresponding pixel on the imaginary image plane in the 3D space, parametered by $\mathbf{r}(t)=\mathbf{o} + t\mathbf{d}$. The color of the ray, $C(\mathbf{r})$, is determined by a weighted integral of colors $c(\mathbf{r}(t), \mathbf{d})$ along every point of the ray, $C(\mathbf{r}) = \int_0^{+\infty}w(t)c(\mathbf{r}(t), \mathbf{d}) \mathrm{d}t.$ Note that in practice, the upper bound of integral is not infinity but rather a fixed big value. This makes sense because it is expected that there is nothing of interest far away from the camera.

In the volume rendering framework, the assigned color weights $w(t)$ of each point on the ray is determined by the volume density $\sigma(t)$ of the point and the accumulated transmittance $T(t)$ at this point, $w(t) = T(t)\sigma(\mathbf{r}(t)).$

The accumulated transmittance $T(t)$ at point $\mathbf{r}(t)$ can be construed as the possibility that a ray, starting from its origin (camera position $\mathbf{o}$) and traversing along the ray path, could arrive at this point without being hindered somewhere before. $T(t)$ is formulated as $T(t) = \exp\left\lparen-\int_0^t\sigma(\mathbf{r}(u))\mathrm{d}u\right\rparen.$

The color $c(\mathbf{r}(t), \mathbf{d})$ is predicted by a neural network. But in contrast to NeRF where the neural network directly predict the volume density $\sigma(\mathbf{r}(t))$, NeuS predicts the signed distance value at the point $f(\mathbf{r}(t))$ instead. NeuS further leverages a mapping to map the signed distance $f(\mathbf{r}(t))$ to the volume density $\sigma(\mathbf{r}(t))$ at this point. The tight mapping of the signed distance field and the volume density distribution enables high fidelity SDF learning through back propagation.

When tying the object geometry with volume rendering, NeuS puts forward two major requirements for the color weight function $w(t)$: unbiased and occlusion-aware. For the rendering to be "unbiased", the color weight function $w(t)$ should attain locally maximal value at the surface intersection point $\mathbf{r}(t_0)$, which, in the opaque scenario, corresponds to the zero level set of the SDF, i.e., $f(\mathbf{r}(t_0)) = 0$. NeuS proposes a mapping from distance to density that is unbiased in the first order. For the rendering to be occlusion-aware, whenever $0 < t_1 < t_2$ and $f(\mathbf{r}(t_1)) = f(\mathbf{r}(t_2))$, $w(t_1)$ should be greater than $w(t_2)$. This aligns with the everyday experience that the object nearer to the camera ($t_1$) should occlude the one ($t_2$) behind it. An overview of the distance-to-density mapping function proposed by NeuS can be found in Appendix A (Eqn. (6) and Eqn. (7)).

We use $\alpha$ to denote the rendered opacity. The $\alpha$ of a pixel is defined as the integral of all color weights $w(t)$ along the corresponding ray, $\alpha = \int_0^{+\infty} w(t) \mathrm{d}t.$

Q: The presentation of Theorem 1.

A: We appreciate your feedback and revise Theorem 1 as follows:

Theorem 1: Consider a ray $\mathbf{r}(t)$ intersecting a plane at a point $\mathbf{r}(t_0)$. Let $f(\mathbf{r}(t))$ be the learned distance function and $w(t)$ the rendering color weight. If $f(\mathbf{r}(t))$ attains its minimum at $t=t_m$ and $\alpha$ is the plane's opacity:

• If $\alpha<0.5$, then $t_0=t_m$, $f(\mathbf{r}(t_m))>0$, and $w(t)$ peaks at $t_m$.
• If $\alpha \geq 0.5$, then $f(\mathbf{r}(t_m)) \leq 0$ and $f(\mathbf{r}(t_0))=0$, positioning $w(t)$'s maximum at $t$ with $f(\mathbf{r}(t))=0$.

Q: The quantitative evaluation with NeUDF.

A: Please refer to the combined responses to all reviewers for detailed evaluations.

Q: A more polite way to contextualize would be to say that $\alpha$-Surf is a concurrent work, without judging its results.

A: Thank you for the guidance. We will revise our discussion to reference $\alpha$Surf as a concurrent work. As mentioned by reviewer ruwJ, $\alpha$Surf has recently released a preliminary version of its code, which we have evaluated and discussed in Figure 1 of the supplementary PDF. We observed that $\alpha$Surf outperforms NeuS and NeUDF+DCUDF on the synthetic dataset, judged by the average Chamfer distances. We hypothesize that the observed surface noise in $\alpha$Surf's results arises from the use of alpha shapes [1] to generate meshes from point clouds. Given the extensive research on producing more reliable meshes from point clouds [2][3][4][5], we anticipate future improvements in $\alpha$Surf results. Given that the currently released code of $\alpha$Surf is preliminary, we regard our existing comparison with $\alpha$Surf as initial findings.

[1] Edelsbrunner et al. On the shape of a set of points in the plane, IEEE Transactions on Information Theory, 29 (4): 551–559, 1983

[2] Zhou et al. Learning a more continuous zero level set in unsigned distance fields through level set projection. ICCV. 2023: 3181-3192.

[3] Liu et al. Ghost on the Shell: An Expressive Representation of General 3D Shapes. ICLR, 2024.

[4] Chen et al. Neural dual contouring. ACM Transactions on Graphics (TOG), 2022, 41(4): 1-13.

[5] Wang et al. Hsdf: Hybrid sign and distance field for modeling surfaces with arbitrary topologies. NeurIPS, 2022, 35: 32172-32185.

The mathematically rigorous proof of the theorem is in the technical appendices of our original submitted paper. Since NeuS is a well-established work in the field, we assume familiarity among our readership. We believe that the fewer symbols the easier it is to understand, so the theorem was originally presented in plain language in our original paper. As requested by the Reviewer GRsP, we revise the theorem symbolically, and present it in a mathematically rigorous way. Here we also outline the proof briefly here for clarity:

(1) Opacity-Distance Relationship: Utilizing the volume rendering equation and NeuS's distance to density function, we establish a correlation between opacity and learned distance. Specifically, we demonstrate that opacity $\leq 0.5$ when the learned distance reaches a non-negative local minimum, and opacity $> 0.5$ when the local minimum is negative.

(2) Color Weight Maximum: We analyze the derivatives of the color rendering weight relative to the ray parameter $t$. We establish that the maximum color weight coincides with the local minimum distance when it is non-negative, and with the zero iso-surface when the local minimum distance is negative.

Reviewer GRsP raised concerns about what he/she described as "ambiguous" concepts, including rendering weights and unbiasedness. Notice that the rendering weight $w(t)$ is fundamental to the volume rendering equations used in prominent models such as NeRF and NeuS. Similarly, "unbiasedness" is a pivotal concept in NeuS, now considered standard in neural rendering of implicit surfaces. Since these terms are well-established within the field, we chose not to redefine them in our initial submission. Recognizing that these concepts might be challenging for readers who are not familiar with NeuS, we plan to include a detailed overview of NeuS in the revised manuscript, as indicated in our rebuttal.

We respectfully disagree with Reviewer GRsP on the notion that machine learning papers neglect rigorous theorem treatment. Contrary to this view, all other reviewers have acknowledged the importance of our theoretical contribution. As pointed by Reviewer 6LF5, our paper "provides a rigorous theoretical analysis, extending the unbiasedness proof of NeuS to cover a range of material opacities from fully transparent to fully opaque. This theoretical work completes and expands the framework of NeuS."

Beyond theoretical contributions, our work includes practical advances. We have developed an algorithm that outperforms existing methods and the concurrent $\alpha$Surf approach. We also introduce a real-world dataset featuring semi-transparent objects, showcasing the practical effectiveness of our approach on these realistic scenes.

The main claims of the paper are theoretical, the entire Method section is based on Theorem 1. When the authors organize the paper that way, they should be aware of the burden of rigorous Math checking in the review. As I said before, Theorem 1 was not stated as a theorem in the original manuscript, making it impossible to check. It was highly ambiguous, with several core definitions missing. My original rating was Strong Reject because that is a major technical and methodological flaw, turning it impossible to adequately evaluate the paper. I turned it a Reject because I had hope the other reviews and rebuttal could point out other perspectives that could make me reevaluate the rating. After carefully reading them and reflecting, that was not the case.

I appreciate that the authors provided a new statement for the theorem, however it is not possible for me to rigorously do that checking in the rebuttal discussion time frame because I have other five reviews in NeurIPS, and submissions. Unfortunately, there was nothing the authors could have done to increase the rating, given the situation. I believe it is risky for NeurIPS to accept a theoretical paper without properly checking the core theorem.

I maintain my advice that the authors should create a new version based on a rigorously stated and proven theorem with adequate context or change the paper structure to not focus on theoretical claims. For example, the original NeuS has a different structure, with the theorem not being protagonist in the claims. Even though I believe every theorem should be rigorously checked (even the ones in supplementary), there is a sad tendency in Machine Learning papers to not treat theorems with proper attention. In some extent, I consider that a disrespect to the field of Mathematics and to mathematicians. Unfortunately, the manuscript crossed the boundary of what is acceptable in a theoretical standpoint.

We thank the reviewer for the comments.

Q: Results and discussion on opaque models (DTU).

A: We have added additional results on the DTU dataset in Figure 4 of the supplementary PDF. The results of our $\alpha$-Neus and the original NeuS are similar.

We show the absolute values of the SDF learned by NeuS, in which DCUDF performs optimization. We observe some local minima/maxima within the model in the distance field. Most of these local minima/maxima appear to be above $r$ and far away from the surface. Local minima larger than $r$ don't affect DCUDF. Thus the influence of local minima is small. In addition, DCUDF can resolve mesh vertices more accurately than Marching Cubes at the same resolution. This has been demonstrated by DCUDF -- see Table 3 of DCUDF paper -- resulting in better Chamfer distance values in some cases.

Q: More explanations about the optimization in Section 3.2.

A: Please refer to the combined responses to all reviewers for detailed evaluations.

Q: The normal maps in Figures 4 and 6.

A: The normal maps displayed in Figures 4 and 6 were generated using the official NeuS codebase for calculating normal maps. We will specify this clearly in the revision.

Q: The setting of NeuS.

A: Please refer to the combined responses to all reviewers for detailed evaluations.

Q: The role of $s$ in Equation (1).

A: In NeuS, $s$ is a learnable parameter that gradually converges to a large value over the course of training. A larger $s$ value sharpens the edges and faces in the reconstructed model, enhancing overall quality. However, in practice, $s$ cannot increase indefinitely due to numerical computation constraints, such as the number of sample points. This caps $s$ at relatively high but finite values, which allows non-zero distance values to influence color calculation along the ray. Colors are derived from the the weighted sum of sampled radiance at points along the rays, and different distance minimum values contributes to achieving different levels of opacity.

We believe that the tendency of a larger $s$, combined with the color loss and the actual situations with MLP and numerical calculation will achieve a balance that leads to the best results.

We thank the reviewer for the comments.

Q: The evaluation set is relatively small, adding more synthetic cases would be fine.

A: Thank you for your suggestion. We added a synthetic example in Figure 3 of the supplementary one-page PDF file as a test of an empty transparent object without refraction and reflection.

Q: Is it possible to include participating media in your pipeline? e.g. if there’s some smog inside a jar, will the unbiasedness still apply? Or can the pipeline be generalized into a stochastic geometry perspective?

A: Including semi-transparent media such as smog within a jar could cause the learned distances to oscillate near zero, simulating the presence of smog. The oscillation might result in multiple local minima, complicating surface extraction. If these local minima could be consistently excluded from the DCUDF (e.g., if the minima are all greater than the user-specified iso-value $r$), the intended surface could potentially be extracted. Theoretically, it is feasible to reconstruct the jar by minimizing smog interference, though practically, this approach may not be robust.

Q: If possible, can GT of real-world test cases be constructed with some "destroy" to the object, like what NeRO did?

A: Thank you for this insightful suggestion. However, capturing transparent objects with an RGBD camera poses challenges due to their optical properties. We will explore this possibility in future research endeavors.

We thank the reviewer for the comments.

Q: Related work on transparent objects.

A: Thank you for highlighting several pertinent studies that we had initially overlooked. We will incorporate a discussion of these works in our revised manuscript. The works [1,2] address the reconstruction of fully transparent objects by separating rendering from geometry or considering the variations in rays due to refraction and reflection. Sharma et al. [3] introduces multiple intersections during the rendering process through an additional network. It primarily focuses on efficient rendering rather than geometric reconstruction.

References:

[1] Wang D, Zhang T, Süsstrunk S. Nemto: Neural environment matting for novel view and relighting synthesis of transparent objects. in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2023: 317-327.

[2] Deng W, Campbell D, Sun C, et al. Ray Deformation Networks for Novel View Synthesis of Refractive Objects. in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2024: 3118-3128.

[3] Sharma G, Rebain D, Yi K M, et al. Volumetric Rendering with Baked Quadrature Fields. ECCV, 2024.

Q: Testing datasets with only transparent objects.

A: As mentioned in the limitations section, we "model only thin transparent objects that have as little refraction as possible". As an extension of NeuS, we primarily discuss NeuS's capability in reconstructing transparent objects. However, since refraction is not considered in NeuS, it is infeasible to use NeuS directly to reconstruct thick transparent objects with strong refraction and reflection. The related works, e.g., NEMTO, reconstruct thick transparent objects with significant refraction and reflection, whose goals are different from ours. We are glad to explore using works like NEMTO as a new backbone to extend the versatility of our method in future work.

Meanwhile, we attach a result of a synthetic empty transparent object without refraction and reflection in Figure 3 of the supplementary one-page PDF file.

Q: Checking the code of $\alpha$Surf.

A: Thank you for the reminder. As of our submission date, $\alpha$Surf had not yet released its source code. We are keen to perform experimental comparisons with $\alpha$Surf and have already conducted some preliminary experiments. The results are detailed in Figure 1 of the supplementary PDF file provided.

We notice that the $\alpha$Surf repository is currently marked as "a preliminary repo for submission of supplementary material", indicating that it may not contain the final release code. Consequently, the results from these experiments should be viewed as preliminary.

Q: Illustration for Equation 8.

A: We added the illustration for Equation (8) to enhance clarity and understanding. Please refer to Figure 5 in the supplementary PDF file. (a) The two red curves represent the two cases of the distance functions when $m\geq 0$ and $m<0$ respectively. (b) is an illustration showing the relationship of the distance $d_0$, $t_0$ and $\cos(\theta)$.

Q: Clarification of training hyperparameters.

A: Please refer to the combined responses to all reviewers for detailed evaluations.

Q: The quantitative evaluation with NeUDF.

A: Please refer to the combined responses to all reviewers for detailed evaluations.