AniSDF: Fused-Granularity Neural Surfaces with Anisotropic Encoding for High-Fidelity 3D Reconstruction

Thank you for your comments and please see our response to the feedbacks below.

Response to W1: AniSDF achieves high-quality geometry reconstruction by fused-granularity structure and blended radiance field instead of solely increasing the maximum hashgrid resolution. Compared to Neuralangelo which set 4096 as the maximum resolution and $2^{22}$ as the maximum hash entry, we use hash-grid resolution from 32 to 2048 and entry as $2^{19}$, a lower resolution but a better performance. Our choice of 2048 hash-grid resolution mainly follows previous work Instant-NSR [1].

To solve your concern about different resolution, we also conduct an ablation study of our resolution choice. In Fig.7, we freeze the minimal resolution as 32 and vary the maximum resolution from 1024 to 4096. It can be seen that by using 1024 as the maximum resolution, the model can not reconstruct some geometric details like the net of the sails. On the other hand, using 4096 as the maximum resolution can achieve slightly better results but requires larger memory consumption (32GB). Therefore, we use 2048 as the default resolution in our model (24GB).

Response to W2: Since most methods tend to learn a solid geometry for those objects with hollow parts, e.g. Mic., Chamfer Distance is not always accurate to reflect the geometry quality. Due to its points sampling process on the mesh, it tends to produce better quantitative results for a smooth surface for these objects. As shown in Fig.10, our reconstructed geometry displays better visual effect than NeRO but gets a worse quantitative result. We also conduct an experiment by adjusting the hyper-parameters to produce a smoother surface and achieve the best metric.

Response to Q1: We extend our ablation study (Fig.6) of the fused-granularity neural surfaces in Fig.8. Results with two fine branches with the same resolution setting produce a noisy surface and require larger memory consumption (30GB), while those with two coarse branches fail to reconstruct geometric details like the net of the sails. In contrast, the fused-granularity structure make the most of both branches and reconstruct the best results with moderate memory consumption (24GB).

[1] Fuqiang Zhao, Yuheng Jiang, Kaixin Yao, Jiakai Zhang, Liao Wang, Haizhao Dai, Yuhui Zhong, Yingliang Zhang Minye Wu, Lan Xu, Jingyi Yu. Human Performance Modeling and Rendering via Neural Animated Mesh. In SIGGRAPH Asia, 2022.

We sincerely thank you for the constructive feedbacks and raising the final rating.

Thank you for your comments and please see our response to the feedbacks below.

Response to Q1: We reproduce the results using the hyperparameter settings in the official Neuralangelo code repository and use the official implementation when training. In fact, Neuralangelo requires a lot of parameter-tuning and it has not provided all optimal configs for each case. A similar result to our paper is also presented in Table.1 of UniSDF [1]. On the contrary, our method does not require complex parameter-tuning and achieve accurate results.

Response to Q2: We have presented the visualization of the surface reconstruction process of Neuralangelo in the rebuttal video. Since geometry learning is supervised only by RGB color, as mentioned in 3.2, the hash grid would first learn the overall shape and try to learn thin structures in the later process. However, the plain coarse-to-fine hashgrid do not have spare power to learn the geometric details, thin structures are then discarded.

To overcome this obstacles, increasing the maximum resolution and maximum hashgrid feature channel can lead to better results as proved by Neuralangelo. Our method on the other hand solve the problem without increasing these parameters and utilize the fused-granularity structure with the blended radiance field to provide an alternative solution that requires less computational resources than Neuralangelo with better results and faster training.

Response to Weakness: Real-time rendering and computation cost are indeed two major weaknesses of AniSDF and we have already mentioned them in the Limitations section.

[1] Fangjinhua Wang, Marie-Julie Rakotosaona, Michael Niemeyer, Richard Szeliski, Marc Pollefeys, Federico Tombari. UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections. In NeurIPS, 2024.

Thank you for your comments and please see our response to the feedbacks below.

Response to W1: Thanks for the wonderful suggestion of the references, we have added these references into our latest version.

Response to W2: Thanks for your advice. We have conducted an ablation study to compare the performance of both branches in coarse (coarse and coarse) and both branches in fine (fine and fine) with that of our current setting (coarse and fine). As shown in Fig.8, the results of two fine branches are noisy and fuzzy, particularly on some smooth surfaces, while two coarse branches cannot reconstruct thin structures, e.g., net of the sails. This is also consistent with our statement in 3.2 of the original paper. We also compare the overall training time of the same iterations, setting both branches to coarse requires 1h40min and both branches to fine requires 3h20min. The fused-granularity structure (coarse+fine) requires about 2h20min to converge.

Response to Q1: Thanks for the careful checking, we have modified the typo.

Response to Q2: Spec-gaussian is another great concurrent work incorporating ASG to learn specularity. Our work is originally motivated by NRFF [1]. We are motivated by the ASG’s ability to capture the anisotropy. In our work, we then use ASG encoding to disambiguate view-dependent effects such as specularity for better surface reconstruction. It enables learning accurate geometry, while the motivation of spec-gaussian falling in the scope of obtaining better rendering. We have added the reference of spec-gaussian in our manuscript.

Response to Q3: Our method mainly focus on reconstruction instead of material decomposition. The reason we use $c_{view}$ and $c_{ref}$ is to disambiguate the geometry from the reflective appearance. Since our method can be considered as a prior-free method, we do not introduce such materials priors in our model and the process of decomposing the scene is fully done by exploiting the power of the texture network with ASG encoding. So $c_{view}$ does not equal to the well-known diffuse color and $c_{ref}$ does not equal to the well-known specular color.

However, $c_{view}$ can be viewed as the approximation of diffuse and $c_{ref}$ as the approximation of specular. Our work then can be considered as disentangling the light field to only the "diffuse" field or "specular" field. For real-world scenes, one scene is never fully diffuse or specular, our method obeys such rules and try to split the into $c_{view}$ and $c_{ref}$. For synthetic scenes, one scene can be fully diffuse of specular, then the weight would be leaning to 1 and obtain a value like 0.999.

Response to Q4: NRFF [1] incorporates (diffuse + tint*specular) when rendering, and it utilizes an additional channel to estimate the value of tint. In the experimental setting of NRFF [1], it utilizes a neural network with more layers and larger hidden layers dimension for optimization. However, since we want to reconstruct high-quality surface in a comparable training with moderate computational resources, our method have smaller neural network. Learning the value of tint may cause additional pressure on the texture network. Therefore, we split the learning of this value by introducing a small network that optimizes the weight, where the radiance fields continuously determine and use the most adapted parametrization for each surface area.

[1] Kang Han, Wei Xiang. Multiscale tensor decomposition and rendering equation encoding for view synthesis. In CVPR, 2023.

Thank you for your comments and please see our response to the feedbacks below.

Response to W1: We have added two novel view synthesis results in the rebuttal video from the shiny blender datasets, it can be seen that we can reconstruct specularity and normal well.

Response to W2: We have provided the visualization of $c_{view}$ and $c_{ref}$ for reference. As shown in Fig.11, our methods can well separate reflectance from base color.

Response to W3: The coarse grid focuses more on the overall structures and tends to discard the details in the early stage. Compared to the overall structure of ship, the net of the sails occupies a very small proportion in the renderings, and thus setting both branches to coarse fail to reconstruct the net as shown in Fig.8. In contrast, the fine grid pays more attention on the geometric details and may overlooks the global shape, so setting both branches to fine may induce noise on smooth surface.

As shown in the rebuttal video, sometimes thin structures that are already discarded in the early stage cannot be restored in the following stage. Of course in the context of plain multi-resolution hashgrids, this can be avoided by increasing the resolution or the block size infinitely to utilize the learning power, which would lead to much higher computational recourses and longer training time like Neuralangelo. Our method on the other hand use fuse-granularity as an alternative to avoid such scaling problems.

Response to Q1: Thanks for the wonderful suggestion. Since AniSDF focuses on surface reconstruction, decomposing the diffuse and specular color is not the primary task. The goal of our task is to reduce the ambiguity brought by the appearance when reconstructing geometry from multi-view images. In this case, the disentanglement process can be viewed as a classifying problem, that is, we want to guide the texture network learn whether the specific surface area belongs to “diffuse” or “specular”. Following the reparametrization introduced by RefNeRF, the reflective direction works well as a guidance of specularity. Therefore, we choose the other classification guidance to be the input view direction d, as the differences between two directions are already sufficient for “diffuse”-“specular” separation.

Setting $c_{view}$ as view-independent is an insightful idea. We have conducted experiment by omitting the $d$ term in the MLP input to make $c_{view}$ view-independent. As shown in Fig.13, the decomposition has subtle differences and with $d$ as an additional input, the model can achieve slightly better PSNR results (26.98 vs. 26.84).

Thank you for your comments and please see our response to the feedbacks below.

Response to W1(Q1): Larger-scene scale experiments can be shown in the Appendix A.2 Unbounded Scenes section where we use MipNeRF360 as the dataset. To further demonstrate AniSDF’s ability to reconstruct reflective surfaces, we have also provided the reconstruction results on NeRO (Coral) and RefNeRF-Real (Sedan and Gardenspheres) as results in Fig.12.

Response to W2(Q2): We have provided the visualization of $c_{view}$ and $c_{ref}$ for reference. As shown in Fig.11, our method achieves great results on separating reflectance from base color.

Response to W3(Q3): We derive our normal as the gradient of the signed distance field: $\mathbf{n}=\nabla d(\mathbf{x}) /\|\nabla d(\mathbf{x})\|$ like most of the other SDF-based methods. As for the mesh extraction, we use marching cubes as the techniques following the NeuS pipeline. We would add these details later in the methods section of the manuscript.

Response to W4(Q5): The choice of $l$ is to follow the implementation of Instant-NSR [1], and we choose $m=10$ to be right in the middle of the beginning level 4 and the ending level $l=16$, where we want the parallel structures each contain the same levels.

Though this is an empirical hyper parameter setting, we also conduct an ablation experiment on the grid resolution. As shown in Fig.9, by setting $m=9$, some geometric details are omitted when learning and by setting $m=11$, some smooth surface can not be reconstructed. This can be interpreted as another demonstration of the observations in 3.2.

Response to Q4:

We have added the comparison of the surface reconstructed method on the training efficiency and memory footprint. It can be seen that with comparable memory consumption, our method requires the leasting training time while obtaining the best results.

[1] Fuqiang Zhao, Yuheng Jiang, Kaixin Yao, Jiakai Zhang, Liao Wang, Haizhao Dai, Yuhui Zhong, Yingliang Zhang Minye Wu, Lan Xu, Jingyi Yu. Human Performance Modeling and Rendering via Neural Animated Mesh. In SIGGRAPH Asia, 2022.

I appreciate the authors for the experiments and visualizations added to the manuscript. Part of my concern is addressed, and I have raised my final rating.