ND-SDF: Learning Normal Deflection Fields for High-Fidelity Indoor Reconstruction

The reviewer acknowledges the author's effort. All of my concerns have been addressed. The reviewer is glad to see the enhancement of this paper and decide to raise the rate from 5 to 6 and raise the confidence.

Congratulations on the improvement.

other questions

what is the dataset conducted in the ablation study

The ablation experiments in the main context are both conducted on the ScanNet++ dataset. Specifically, module ablation is performed on scene 036bce3393, and prior model ablation is carried out on scene 0e75f3c4d9 of ScanNet++.

The number of selected scenes in each dataset for the experiment.

• For ScanNet, four scenes are selected, consistent with MonoSDF and DebSDF. We report it in Section 4.1 ScanNet paragraph.
• For Replica, eight scenes are selected, consistent with MonoSDF and DebSDF.
• For ScanNet++, six scenes are selected. The specific scene ids are summarized in Table.10
• For T&T, we tested on the four indoor scenes of the advanced split, consistent with MonoSDF and DebSDF.

"GT depth" and "GT Normal"

Thank you for carefully pointing out this problem! We have updated the framework figure (Figure.2) in the revised version.

MLP or Grid

We have denoted the known scene representations of MonoSDF and DebSDF in the revised version.

We sincerely appreciate your time and effort in reviewing our work. Thank you for your thoughtful comments and constructive suggestions on our paper! Here, we address the concerns raised in the review:

Comparisons with newer methods

Concerns about insufficient qualitative or quantitative comparison results with the latest methods, such as TUVR, and DebSDF.

Additionally, we included DebSDF and TUVR for comparison. As TUVR is not open-sourced and was originally tested only on the DTU dataset, we incorporated TUVR's unbiasing function (Eq.15) into the official MonoSDF (Grid) framework to create the TUVR (MonoSDF-Grid) method, hereafter referred to simply as TUVR. The quantitative results are as follows:

On Replica, we directly use the official evaluation results from DebSDF:

ScanNet++:

For the T&T dataset, to our knowledge, DebSDF was evaluated at 1024 resolution and MonoSDF at 768. To align with DebSDF, we increased the marching cubes resolution from 768 to 1024 and resubmitted it to the official T&T evaluation site. Additionally, we assessed results at a higher resolution of 2048, at which thin structures are extracted. We indicate the known evaluation resolution after the method name. The results are:

ND-SDF outperforms the baseline methods, especially at higher resolutions, indicating its superior ability to capture finer details and obtain more accurate reconstructions.

The specific modifications of this part in the revised version are:

• We update Figure.4, Table.2, Table.3, Table.4.
• We update the corresponding experiment descriptions for these four indoor datasets in Section.4.1.
• We add detailed comparison results on ScanNet++ and T&T in Figure.15, Figure.16 and Figure.17.

We strongly suggest the reviewer to refer to the updated qualitative and quantitative comparison results.

About unbiasing methods

In Figure.3, we illustrate an actual scenario that the bias issue will affect the rendering weight. The main purpose of it is to stress the necessity of employing unbiasing transformation in SDF-based reconstructions. TUVR and DebSDF have presented detailed bias error curves in simpler scenarios, validating that the bias error follows the order: VolSDF ≥ Neus ≥ TUVR ≥ DebSDF. The key contribution of our method lies in identifying and addressing the unbiasing convergence issue, which is thoroughly analyzed in Appendix B.1, supported by both qualitative and quantitative ablation experiments.

We further analyze the current bias issues in Appendix C.4; we strongly recommend that the reviewer refer to it for a more detailed discussion of these issues.

Advantages towards uncertainty based methods

H2OSDF uses a prior model to obtain uncertainty and then re-weights, suffering from limited generalization and robustness.

In contrast to DebSDF's uncertainty field, we directly model the geometry-related SO(3) residuals to capture the deviation between prior geometry and scene geometry. Inherently, uncertainty is a kind of intensity field that is independent of geometry and measures the confidence of monocular cues. The variance term is introduced when using the log-likelihood loss, which is unrelated to geometry, may cause the model to focus on variance changes rather than the underlying data patterns.

Our approach introduces a deflection term that is highly correlated with geometry, accurately reflecting deviations. DebSDF requires careful design of a variance threshold and tuning of numerous hyperparameters to prevent training instability, which may affect its generalization in complex scenes. We avoid this issue; the only aspect requiring special design in our method is the modulation function (Figure.8). Setting 𝑔(15°)=0.5 follows an intuitive design and yields promising results.

Mask comparisons and sampling strategy discussion

We visualize the masks obtained by various methods that have been mentioned in the review in Figure.12. And we provide a comprehensive analysis in Appendix B.3. They are now available in the revised version.

Trainig objective

For smooth regions, priors assist in rapid convergence during the initial warm-up phase. The designed $L^{ad}_{normal}$ loss provides a flexible capability, allowing the model to gradually converge in the right direction to fit the photometric consistency, even under supervision from incorrect priors. Since the gradients of modulation functions are detached, this indicates that the model autonomously learns the angular deviations of structural regions to converge correctly.

We are grateful to hear our feedback has addressed all of your concerns. It would be appreciated if you could recommend accepting our paper in the reviewers-pc discussion. We thank you again for your effort in reviewing our paper.

Best regards,

ND-SDF Authors

We are grateful to hear our feedback has addressed your major concerns. It would be appreciated if you could recommend accepting our paper in the reviewers-pc discussion. We thank you again for your effort in reviewing our paper.

Best regards,

ND-SDF Authors

Thank you for the constructive comments and the valuable suggestions. Here, we address the concerns raised in the review.

Comparisons with newer methods

Concerns about insufficient qualitative or quantitative comparison results with latest methods, such as TUVR, DebSDF.

Additionally, we included DebSDF and TUVR for comparison. As TUVR is not open-sourced and was originally tested only on the DTU dataset, we incorporated TUVR's unbiasing function (Eq.15) into the official MonoSDF (Grid) framework to create the TUVR (MonoSDF-Grid) method, hereafter referred to simply as TUVR. The quantitative results are as follows:

On Replica, we directly use the official evaluation results from DebSDF:

On ScanNet++, our method also achieves the best results:

For the T&T dataset, to our knowledge, DebSDF was evaluated at a 1024 resolution and MonoSDF at 768. To align with DebSDF, we increased the marching cubes resolution from 768 to 1024 and resubmitted it to the official T&T evaluation site. Additionally, we assessed comparative results at a higher resolution of 2048, at which more complex and thin structures are extracted. We indicate the known evaluation resolution after the method name. The results are:

ND-SDF outperforms all baseline methods, especially at higher resolutions, indicating its superior ability to capture finer details and obtain more accurate reconstructions.

The specific modifications of this part in the revised version are:

• We update Figure.4, Table.2, Table.3, Table.4.
• We update the corresponding experiment descriptions for these four indoor datasets in Section.4.1.
• We add detailed comparison results on ScanNet++ and T&T in Figure.15, Figure.16 and Figure.17.

We strongly suggest the reviewer to refer to the updated qualitative and quantitative comparison results.

Lack of citation

We clarify that our method is entirely independent of NC-SDF. Although we similarly model an SO(3) residual field to enhance surface reconstruction, our design diverges substantially. NC-SDF employs a two-stage approach, yet struggles with noise in the second stage’s explicit deflection. In contrast, our adaptive prior loss automatically assigns weights based on sample characteristics (Eq.12), removing the need for staged training. Additionally, NC-SDF requires extra boundary information for optimization, reducing robustness and adding hyperparameters. While NC-SDF was tested on datasets with simple layouts like ScanNet and ICL-NUIM, we evaluated our method on more complex indoor scenes, including ScanNet++ and T&T. Extensive experiments confirm our method’s superior generalization in challenging environments, an advantage not observed in NC-SDF.

We updated this discussion in Appendix C.4 in the revised version.

*Optional: Confidence map comparisons

We visualize the masks obtained by various methods in Figure.12, including edge mask(EaNeuS), NCC mask(Equation 6 in NeuRIS), and uncertainty-aware mask(DebSDF). And we provide a comprehensive analysis in Appendix B.3. They are now available in the revised version.

Thank you for your thoughtful feedback. Below, we address the concerns raised in the review.

Convergence issue

We found that uniformly applying the unbiasing transformation (Eq. 15) can cause convergence issues. To address this, we propose a partial unbiasing strategy, or deflection angle-guided unbiased rendering (Eq. 27).

This issue has been analyzed in Appendix B.1 in the original paper. And we present both qualitative and quantitative ablation experiments to validate it. We believe the reviewer may have missed the relevant information.

Besides, we additionally visualize the inverse variance (1/β) of the SDF-density conversion function (Eq.3) during training and find that, without partial unbiasing, 1/β fails to converge to a relatively high value, indicating inaccurate and blurry surfaces. The variance curves and comprehensive analysis are presented in the revised PDF version; reviewers can refer to Appendix B.1 for details.

We add a very detailed discussion in Appendix C.4 about the unbiasing method and the motivation of the partial unbiasing strategy. If the reviewers are interested, refer to this section for details.

*Optional: More comparison results

To solve some reviews' concerns about insufficient qualitative and quantitative comparison results with newer methods. We add the latest state-of-the-art method DebSDF for comparison on all these datasets in the in the revised version, including ScanNet, ScanNet++, Replica and T&T datasets .

The specific modifications of this part are:

• We update Figure.4, Table.2, Table.3, Table.4.
• We update the corresponding experiment descriptions for these four indoor datasets in Section.4.1.
• We add detailed comparison results on ScanNet++ and T&T in Figure.16, Figure.17 and Figure.18.

We demonstrate the superiority of our method to capture accurate detailed structures, even in large-scale or complex indoor scenes. The updated comparisons prove the better generalization abilities of our method than DebSDF.

*Optional: Confidence map comparisons

We visualize the masks obtained by various methods in Figure.12, including edge mask(EaNeuS), NCC mask(Equation 6 in NeuRIS), and uncertainty-aware mask(DebSDF). And we provide a comprehensive analysis in Appendix B.3. They are now available in the revised version.

Answer to questions

In Line 280, the $\theta$ is correctly changed to $\Delta\theta$. Thank you for carefully pointing out this problem!

We hope our reply could address your questions. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper. We thank you again for your effort in reviewing our paper.

Best regards,

ND-SDF Authors

I thank the author for the detailed rebuttal. I do not have more questions. I'll maintain my score.

Thank you for the thoughtful comments and constructive suggestions on our paper. Below, we address the concerns raised in the review.

Comparisons on DTU Dataset

ND-SDF's primary advantage is its capacity to handle scenes with complex layouts, encompassing both richly textured structures and large, smooth, low-textured areas. Generally, introducing blurred priors can reduce reconstruction accuracy, especially for textured objects or outdoor scenes. While our pipeline is not specifically tailored for single objects, we validated it on the DTU dataset without specifically adjusting parameters and reduced training steps from 128,000 to 50,000, yet still achieved promising results:

Notably, MonoSDF uses geometric priors only in the early training stages for quick initialization, avoiding potential accuracy loss from incorrect priors. In contrast, our approach consistently applies priors throughout the entire training process.

Noise and artifacts

There are indeed issues with noise and artifacts. From our opinion, the main cause of this is the poor quality of the ScanNet dataset and the use of hash grids for scene representation. The ScanNet dataset is captured by an iPad with an additional structure sensor. The images suffer from relatively low resolution, motion blur, and lighting variations. These poor-quality images led to significant perspective inconsistencies, which had a considerable impact on hashgrids, as they are locally structured for scene representation. Using an MLP would easily smooth out these issues, but in order to preserve as much detail as possible, we chose to use grids uniformly. Another cause are the non-uniform re-weight strategy (Section 3.4) guided by the deflection angles and the use of unbiasing method. But we believe their impact is minimal and controllable.

To solve some reviews' concerns about insufficient qualitative and quantitative comparison results with newer methods. We add the latest state-of-the-art method DebSDF for comparison in the revised version. We add detailed comparison results on ScanNet++ and T&T in Figure.15, Figure.16 and Figure.17.

The reviewer could refer to these new comparison results to check out the noise issue, by comparing our method with MonoSDF and DebSDF. If there are still concerns about significant artifacts, could the reviewer provide more detailed descriptions, such as which dataset, which figure, and the specific areas of concern?

About the possible solutions of this problem, we have tried smooth term proposed in Permute-SDF and it has minimal improvement.

Other questions

normal prior problem seems only solved half

As stated in Section 3.3, we propose the $L_{normal}^{ad}$ (Eq.12) loss to adjust the utilization of various priors. We increase the weight ($g^d(\Delta\theta)$) of the deflected term (Eq.10) in structural regions to encourage the model to capture geometric deviations. In contrast, the weight ($g(\Delta\theta)$) of the naive normal prior term (Eq.5) decreases to prevent incorrect priors from interfering with fine-structure generation. In smooth regions, the deflected term's weight approaches zero, allowing the priors to fully guide the recovery of flat areas. Ideally, the adaptive loss ($L^{ad}_{normal}$) converges to zero. We consider this issue is well addressed. The reviewer can refer to Figure.8 for the $g^d,g$ curves.

An implausible assumption: smooth area the prior is relatively accurate.

We believe this assumption is reasonably reliable. The monocular cues are generated by a pretrained neural network, which has a strong smooth inductive bias. Generally, smooth regions exhibit similar and simple feature patterns, enabling the network to more effectively capture and predict monocular cues.

Concerns about the poor ability of SDF methods to capture thin-structures

We agree with this opinion since SDF methods inherently can not achieve very high-quality reconstructions, especially in large-scale indoor scenes with large-area smooth regions. But we still achieve remarkable reconstruction quality, balancing the smoothness and detail. The reviewer can refer to qualitative results in Appendix C for details (Figure 14-Figure 18).

Supervising using depth priors

If the reviewer refers to designing an ad_depth structure similar to ad_normal, we believe this could introduce unnecessary noise into the normal deflection field learning. A more feasible approach might be to design a depth residual field. However, since depth priors lack a scale, the loss function may exhibit significant oscillations, leading to unstable convergence. This requires careful consideration. The ND-SDF learns the general difference between prior and scene geometries to distinguish structured regions, while simultaneously masking the depth priors. The effectiveness of the adaptive depth loss is validated in Appendix B.2.

Thank you for your positive feedback. We are glad that our response has addressed the concerns raised in the review. However, we believe that certain points warrant further experimental validation for greater rigor.

Assumption: Priors of smooth regions are accurate

Concerns about trivially assuming the high accuracy of monocular cues in simple flat regions.

To deeply investigate this issue, we conducted experiments under the assumption that the priors for smooth regions are inaccurate. Specifically, we manually rotated all monocular normals in the smooth regions by 0°~60°. A SAM model was employed to detect the flat areas. The rotation was performed by rotating the normal priors by a specified degree (e.g., 60°) toward a pre-defined axis. We evaluated MonoSDF, DebSDF, and ND-SDF on the 0e75f3c4d scene from ScanNet++. The results are summarized as follows:

As demonstrated above, decreases in F-score are observed across all methods. Specifically, MonoSDF produces an entirely abnormal result, as it rigidly relies on the biased priors. DebSDF fails to maintain robustness in this scenario. Although the uncertainty field in DebSDF can effectively mask irregular priors by learning uncertainty variances, it leads to inefficient utilization of the consistency information inherent in monocular cues within smooth regions. In contrast, our method explicitly models the actual bias amplitude, achieving notably robust performance with the smallest decrease in F-score, even when the normal priors are biased by as much as 60°. These findings further validate the generalization and robustness of our method in non-trivial assumptions of the monocular cues.

We have included this study in Appendix B.4 of the revised manuscript, along with detailed visualization results. The reviewer is encouraged to refer to this section for additional details.

*Optional: More comparison results

To solve some reviews' concerns about insufficient qualitative and quantitative comparison results with newer methods. We add the latest state-of-the-art method DebSDF for comparison on all these datasets in the in the revised version, including ScanNet, ScanNet++, Replica and T&T datasets .

The specific modifications of this part are:

• We update Figure.4, Table.2, Table.3, Table.4.
• We update the corresponding experiment descriptions for these four indoor datasets in Section.4.1.
• We add detailed comparison results on ScanNet++ and T&T in Figure.16, Figure.17 and Figure.18.

We demonstrate the superiority of our method to capture accurate detailed structures, even in large-scale or complex indoor scenes. The updated comparisons prove the better generalization abilities of our method than DebSDF.

*Optional: Confidence map comparisons

We visualize the masks that highlight structural regions obtained by various methods in Figure.12, including edge mask(EaNeuS), NCC mask(Equation 6 in NeuRIS), and uncertainty-aware mask(DebSDF). And we provide a comprehensive analysis in Appendix B.3. They are now available in the revised version.

End

We hope our reply could provide the reviewer with deeper insights into our approach. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wondered if you might still have any concerns we could address. It would be appreciated if you could raise your score on our paper. We thank you again for your effort in reviewing our paper.

Best regards,

ND-SDF Authors

*Optional: Additional details about the normal problem

We truly thank the reviewer for clarifying the normal problem. Actually, this problem can be related to our training objective:

Training objective of ND-SDF: For smooth regions, priors assist in rapid convergence during the initial warm-up phase (Appendix A.2). The designed $L^{ad}_{normal}$ loss (Eq.12) provides a flexible capability, allowing the model to gradually converge in the right direction to fit the photometric consistency and generate accurate geometry, even when supervised with incorrect priors in structural regions. Since the gradients of modulation functions are detached, this suggests that the model autonomously learns the angular deviations of structural regions to converge correctly.

The above indicates that the photometric loss, rather than the suppressed normal loss, is mainly utilized to remedy the structural regions. The specially designed re-weight, deviations-driven sampling, and partial unbiasing strategies are for further boosting the recovery of these regions. Indeed, we designed exactly these strategies to address the half-normal problem. Maybe other high-quality priors can be directly used to guide the optimization of thin structures as the reviewer mentioned.

End

Thank you for your positive comments on our work. We truly appreciate your acknowledgment of our contributions and the raised score. Once again, we thank you for your time and effort in reviewing our paper.

Best regards,

ND-SDF Authors