GeoSVR: Taming Sparse Voxels for Geometrically Accurate Surface Reconstruction

Responses to Reviewer ZuxM

We sincerely appreciate Reviewer ZuxM's time and efforts for the positive comments and thorough suggestions. We notice that the concerns mainly relate to the metrics selection across datasets, analysis about the rendering quality, comparison with the related approach, and the running overhead. Here, we address the questions raised by the reviewer individually:

---

Q1: Metric consistency across DTU, TnT and Mip-NeRF 360 datasets.

• The selection of Chamfer Distance and F1-Score on DTU and TnT datasets.

We agree with the reviewer that adding more comprehensive and consistent metric evaluation across different datasets is extremely important for the community, but it may be particularly difficult to implement currently. In our work, we separately adopt F1-Score and Chamfer Distance for TnT and DTU datasets following the standard settings in previous works with no manual selection to produce fair and persuasive comparisons. We'd like to note that these two dataset-metric bindings are generally considered as the prevailing standard settings for surface reconstruction currently, which have been widely used and accepted by the famous works like VolSDF, NeuralWarp, NeuS, Neuralangelo, 2DGS and etc. for years. Especially, F1-Score is the officially specified metric for the TnT dataset equipped with an official evaluation toolkit; meanwhile, using Chamfer Distance to evaluate DTU datasets with the officially provided distance calculation process has also been a well-recognized tradition in the field of surface reconstruction. Although a unified metric setting would be better, ideally, designing and implementing a fair, objective, and robust evaluation tool of a new metric for a specific dataset from scratch indeed requires a lot of effort and time, which may be out of the scope of this paper.

We fully understand the concern from the reviewer and sincerely appreciate the provided thoughtful suggestions. Currently, considering exhibiting strong performance in the most prevailing and well-accepted standard settings across the two mainstream and challenging datasets TnT and DTU, combined with the rendering quality evaluation on Mip-NeRF 360 and various reported visualization results, we believe the contribution of our work can be well verified and recognized by the provided fair, comprehensive and objective supports in the paper.

• Additional rendering performance on DTU and TnT datasets.

Thanks for your constructive suggestion and thorough consideration. We'd like to express that evaluating rendering performance on the TnT and DTU datasets with the surface reconstruction task is not common in the field, mainly because these reconstruction settings are different from the novel view synthesis task (like the one applied to Mip-NeRF 360 setting), of which the formers use all the captured views as the inputs in the training to reconstruct a surface model for evaluation, therefore there's not an unseen test set of the RGB views left for a fair and objective evaluation. Considering a potential value of justification, following the comments, we provide a comparison here that conducts the rendering quality evaluation on the training set. But we want to note that these results should be interpreted with caution because these data have not been isolated during the training. Therefore, they can not fairly and justly reflect the performance of each method. We sincerely recommend referring to the results in the Mip-NeRF 360 setting for a fair, objective, and persuasive comparison.

Table R1. Rendering Quality on DTU and TnT datasets. (The evaluation can be only conducted on the training set as no test set is available, and should be interpreted with caution.)

---

Q2: Analysis on the novel view synthesis performance.

Thanks for the constructive suggestion! While achieving the core target of high-quality surface reconstruction, the experiments of novel view synthesis in Mip-NeRF 360 datasets show that our method also maintains a strong capability in appearance reconstruction, striking a great balance between these two tasks. Typically, it's a common phenomenon that the rendering quality slightly drops when the model is forced to learn the accurate geometry (e.g., SuGaR, 2DGS, VCR-GauS, PGSR compared to 3DGS). This mainly lies in that the captured multi-view images from the real world do not always retain an ideal photometric multi-view consistency that matches the correct geometry, such as the reflective regions and transparent materials, especially the current radiance fields for surface reconstruction seldom consider complex ray tracing, but mostly only once forward. In that situation, a distorted geometry in a local minimum may be better than the correct one to express an approximately accurate appearance. Especially, in Mip-NeRF 360, these kinds of regions are more common in the Indoor Scenes, like the metal and glasses in Counter, a large reflective television screen in Room, etc. Therefore, our performance in the Indoor categorization is worse than that in Outdoor, and may also perform lower rendering quality than the baseline methods with relatively less accuracy in geometry.

Following the suggestion, these discussions will be added in the revised version, and we'll explore achieving better consistency between the accurate geometry and challenging appearance cases in the future.

---

Q3: Comparison with 3DGS-MCMC.

Following the suggestion, we provide an additional comparison with 3DGS-MCMC here, where GeoSVR still achieves SOTA performance, which further enhances the advancement and significance of our GeoSVR to explore and achieve high-quality surface reconstruction on the under-investigated representation. 3DGS-MCMC makes important progress by reformulating 3DGS as MCMC sampling for optimization, improving the rendering quality and robustness of initialization for novel view synthesis. Considering it was not designed for surface reconstruction initially, we made the evaluation by applying it to the SOTA 3DGS baseline PGSR for fairness. As shown in Table R2, while 3DGS-MCMC relieves the reliance on initial SfM point clouds by enabling random initialization, our method still performs the best in the comparison. Due to the different focus of the tasks, some designs in 3DGS-MCMC, like the noise-driven SGLD, can bring significant improvements in appearance fitting, but may not quite match the goal of accurate surface formation currently.

Table R2. Comparisons with 3DGS-MCMC on TnT dataset. ("SfM" and "Random" denote the initialization type, and "Depth" indicates that Patch-wise Depth loss is adopted. We marked the changes of replacing the initialization from "SfM" to "Random".)

---

Q4: Rendering speed comparison.

Our method keeps a high efficiency in the rendering FPS, outperforming the 3DGS baselines in most cases, and is even slightly faster than SVRaster in practice. In Section E of the supplementary material, we briefly discussed the rendering speed of GeoSVR and its relation to SVRaster. As analyzed, since our method does not introduce any burden during inference, its rendering speed is theoretically the same as SVRaster under the same number of voxels. Here we additionally compare our method with SVRaster and the 3DGS baselines on the three datasets in Table R3, and report the inference speed of our ablation study in Table R4. These results have shown the superior inference speed of GeoSVR compared to various baselines. And by forcing an accurate geometry to be learned, the voxels can be utilized more efficiently by removing distorted surfaces and artifacts, which further improves the advantage of GeoSVR in terms of efficiency.

Table R3. Comparisons on Inference FPS.

Table R4. Comparisons on Inference and Number of Voxels on TnT dataset.

---

Q5: Writing problems.

Thanks for your careful notice and pointing out the problems! We have fixed these errors and checked the paper thoroughly to ensure writing quality. The revisions will be included at the next opportunity to upload the paper.

Responses to Reviewer dj1z

We sincerely thank Reviewer dj1z for the time and effort in the constructive and encouraging review. We found the reviewer’s main concerns are related to how the contributions of various loss terms and runtime overhead. Here, we address the specific questions from the reviewer individually:

---

Q1: Reported visualized effects of only two components in Figures 6 and 7. It's better to provide more visualizations.

Thanks for the constructive suggestion! In our submission, we visualized the ablation results for the three most insightful components in Figures 6 and 7 of the main paper, and Figures 9 and 10 of the supplementary material, along with detailed analysis in Section A.1 and Section A.2. We believe these visualizations can provide comprehensive support for justifying the effects of Voxel-Uncertainty Depth Constraint, Voxel Dropout, and Surface Rectification.

Specifically, 1) for the Voxel-Uncertainty Depth Constraint, Figure 6 in the main paper and Figure 9 in the supplementary material have shown an obvious improvement is brought in constraint the scene geometry; 2) For the Voxel Dropout, due to the space limitation, we reported the visualization in the Figure 10 in the supplementary material and made analysis in the Section A.2; 3) For the Surface Rectification, some visualizations were reported in the Figure 7, just as noticed by the reviewer. Especially, we'd like to briefly explain here that our Surface Rectification is designed for pursuing the challenging target of extremely high-precision surfaces reconstruction. It works mainly in the stages of an almost well-reconstructed geometry to certify the delicate and difficult details, and thus affects mainly in the details that is not as obvious in visualization as the other techniques that aims to reshape a coarsely correct geometry from a worse start, but this does not equals a lower importance, especially considering the raising difficulty and criticalness in accuracy improvement the situation that the reconstruction results have already been very close to the ground truths.

We agree with the reviewer that more visualization results would be beneficial to justify the inclusion of the components, and will add the visualized comparisons for the remaining Scaling Penalty technique in the revised version. Due to the official restriction of the rebuttal in forbidding images, we are sorry that the materials can not be presented immediately, but will be included along with the next chance of paper update.

---

Q2: Performance Gaining of Multi-view Regularization and Patch-wise Depth.

• Explanation and analysis of the performance in the ablation study.

Thanks for your thoughtful feedback! We'd like to clarify that Multi-view Regularization and Patch-wise Depth did not provide the key improvement in performance. From the ablation study in Table 4, it's notable that although Multi-view Regularization and Patch-wise Depth work well, the improvements are based on a weak initial performance and have achieved an upper-bound of 0.538 in F1-Score, which is limited by the insufficient capacity to address the valuable problems our paper has revealed and aimed to solve. Notably, this score is only comparable to the previous SOTA method PGSR with depth prior added, which has not shown the advancement that our GeoSVR achieved. Based on this start, the results from settings of C, D, and E show that notable improvements have been brought by our proposed Voxel-Uncertainty Depth Constraint, Voxel Dropout, Surface Rectification, and Scaling Penalty, and achieve a remarkable 0.560 F1-Score. Combined with our reported visualization results, the critical and irreplaceable effect of each component can be well justified.

Besides notable improvements have been achieved by our technical contributions, we'd like to discuss that the progress made in different cases should be evaluated distinguishably, which correspond to different difficulties, Especially, in the situation where GeoSVR exhibits its advancements, the scenes are already nearly well reconstructed, and the remaining problems are much more difficult and challenging to solve. In this case, the progress we make is particularly critical and important, and is of great value for inspiring the pursuit of higher-precision reconstruction in the future.

We'd like to note that Multi-view Regularization and Patch-wise depth are not the core contributions of our work, which has been clearly described in our paper. These two classic and general techniques are adopted to provide a basic capability to support our advanced research, but do not act as the critical innovations of our work. We express our sincere respect for these great prior works and reported their effects in the ablation study to ensure the transparency, but would like to emphasize that the patch-warping based multi-view consistency is a long-standing effective technique for accurate geometry learning that has been widely used in many previous works for over years [18, 44, 65, 16, 12, 7, 41, 8, 47], which has become a common and standard solution for the field; and we regard the patch-wise depth just as one of an under-investigated loss function in the field of surface reconstruction for our selected monocular depth constraint, which should be well introduced in the paper.

As recognized by the reviewers, the contribution of GeoSVR mainly lies in the proposed Voxel-Uncertainty Depth Constraint, Voxel Dropout, Surface Rectification, and Scaling Penalty. The Voxel-Uncertainty Depth Constraint, which is highly praised by the reviewers, is an insightful and sound way to maximize the effect of monocular depth cues while presenting a voxel-oriented uncertainty to avoid quality degradation. It innovatively analyzes the geometric uncertainty based on the voxel representation, enabling effective and robust scene constraints while preserving highly accurate geometries. Then, we systematically analyzed the problems in voxel-based surface formation, and targetedly proposed the Voxel Dropout, Surface Rectification, and Scaling Penalty in the aspect of multi-view geometry regularization and the bias in surface-voxel alignment. Our investigation of high-quality sparse voxel-based surface reconstruction brings new understanding to the community. With the thorough problem analysis and our insightful solution, GeoSVR is composed and demonstrates strong performance across various benchmarks. And besides the state-of-the-art performance achieved, the revealed problems also bring fresh insights and could inspire more advancements to facilitate the development of the field.

---

Q3: Runtime overhead of Table 4.

Thanks for the constructive suggestion! Here we provided the efficiency metrics of the ablation study Table 4. From the results, it can be observed that the multi-view regularization contributes most to the training time consumption increasing, while the other components maintain high efficiency in all aspects, including the inference FPS, memory overheads, and the required number of voxels. According to our analysis, this is mainly caused by the less efficient code implementation of the regularization, which we plan to solve in the future. The training time of the base SVRaster is a bit different from the official result in Table 1, which is caused by a slight hyperparameter adjustment to obtain an ideal reconstruction accuracy, which is consistent with the details we reported in lines 245-251 and is the same as the other ablation settings. We'll add the results in the revised version following the suggestion.

Responses to Reviewer fLdW

We sincerely thank Reviewer fLdW for the positive evaluation and insightful feedback. We noticed that the reviewer’s main concerns involve analysis of computational overhead, ablation detail for depth constraint, and robustness in failure scenarios. Here, we address the questions from the reviewer individually:

---

Q1: Memory consumption of the proposed method.

Thanks for the constructive suggestion! Our method maintains similar or even less GPU memory overhead compared to SVRaster, and practically is able to reduce over 8% of GPU memory (over 1.1 GB in the original 12.3 GB) in the experiments on TnT datasets on average. Here we supplement the metrics of running overheads corresponding to the ablation study of Table 4 to show the consumption of each component. From the reported results, it can be observed that our proposed components exhibit superior efficiency in GPU consumption, which seldom increases GPU memory occupancy under a similar number of voxels. Especially, this achievement is contributed by our efficiency-focused designs in constraint selection, restrained voxel-level regularizations, and combined with the efficient coding implementation. Moreover, while our proposed techniques effectively enforce a correct geometry to be learned, the artifacts, redundant voxels, and distorted surfaces can be removed or well corrected, therefore, the GPU memory requirement could even decrease along with the number of used voxels reduced. This demonstrates the high efficiency and effectiveness in improving the surface reconstruction quality in the methodology, but not solely an efficiency-quality trade-off. We'll add these results in the revised version to support a more comprehensive justification.

---

Q2&Q3: Comparison with naive depth constraint. & Robustness when the monocular depth prior fails.

Thanks for the feedback from the reviewer. From Table 4 in the main paper and Table 7 in the supplementary material, and the corresponding qualitative results in Figure 6 in the main paper and Figure 9 in the supplementary material, our method, along with the Voxel-Uncertainty Depth Constraint, can be well verified to be effective and robust in various cases with comprehensive validations. Here we provide our analysis.

• Q2: Comparison with naive depth constraint.

In the ablation study of Table 4, we first compared the performance between the two alternatives to apply the depth priors (i.e., Sparse Points and Inverse Depth in the table), and found that our used Patch-wise Depth performs best, which is the foundation for the following ablation settings. To show the impact of our proposed Voxel-Uncertainty Depth Constraint, we directly compared the two situations with or without this technique in Table 4 by Item D and E. As analyzed in Section 4.2, this technique can remarkably recognize the uncertain regions and refine the geometry while preserving the well-reconstructed parts, even though the reconstruction already achieves a high quality. Recognizing the importance of a comprehensive verification, we additionally provided a fine-grained ablation study for the depth constraint in Section A in the supplementary material, including various comparisons on different depth estimators between no depth constraint / without Voxel-Uncertainty Depth Constraint / with Voxel-Uncertainty Depth Constraint. According to the table, our proposed technique consistently delivers obvious improvements in both accuracy and completeness, and outperforms the other situations regardless of the quality of the provided depth priors, which demonstrates its effectiveness and robustness.

Q3: Robustness when the monocular depth prior fails.

Although it's extremely rare to be entirely invalid for the advanced depth estimators that are trained on large corpus, our method still support the capability for high-quality reconstruction with totally no depth priors involved, as verified in Table 7, which shows that our method can still achieves a comparable performance than the previous SOTA method PGSR in such an unusual case. Notably, in this comparison, PGSR still relies on an SfM point cloud as the prior, while our method doesn't require it. Besides this extreme case, a more common challenging situation is that the provided depth prior is of a much lower quality than expected. To verify our performance on this case, we ablate our method in Table 7 by not using the SOTA monocular estimators but the relatively old DepthAnything V1, which is much weaker in accuracy, robustness, and detail prediction than DepthAnything V2 and DepthPro, to stimulate the situation. We find that even in this case, our method can still provide high-quality surface reconstruction, mainly thanks to our Voxel-Uncertainty Depth Constraint that successfully protects the well-reconstructed parts not influenced by the errors in the monocular depth priors, while still applying effective geometry constraints to the uncertain areas. In summary, our method is robust for different failure cases of the depth priors.

To better guide the readers to locate the additional experiments, we'll add clearer reminders in the main paper to the related supplementary material.

---

Q4: Will the Voxel Dropout design encourage larger and more redundant voxels?

Thanks for your insightful question! According to the efficiency ablation results from the table in the response to Q1 and observations like Figure 10 in the supplementary material, our Voxel Dropout will not cause evidently redundant or larger voxels to be produced and harm the reconstruction quality. This robustness can be ensured from two aspects: 1) In our representations, the size of the voxels is strictly controlled to be unidirectional, which can only become smaller by subdivision. Therefore, the Voxel Dropout can be safely applied without the risk of distorting the voxels to fit a larger space by becoming abnormally bigger in size. 2) In the implementation, we have separated the gradient from Voxel Dropout from the accumulated gradient indicator for voxel subdivision. As a result, the incorrect voxels that suffer from large impacts will not unexpectedly divide into a set of redundant small voxels, ensuring the sparsity control of voxels stays controllable but not worsened by the enforced regularizations. We'll add these explanations of Voxel Dropout in the revised version to help the readers better understand our method.

Responses to Reviewer gVE6

We sincerely thank Reviewer gVE6 for recognizing our work and providing valuable comments. We noticed that the main concerns from the reviewer are the performance and explanation of the Surface Rectification design, improvements on the writing and the relation to SVRaster in initialization. Here, we address the questions individually:

---

Q1: Robustness and Effectiveness of the Surface Rectification.

• Will Surface Rectification cause inaccurate shrunk geometry by over-pulling the surfaces to the "out" direction?

Our proposed Surface Rectification aims to address the biased surface problem at voxel level, offering robust correction in the whole optimization process. It can prevent the over-shrunk geometry problem in two aspects: 1) First, while a typical effect of Surface Rectification is to solve the front-biased surface by pulling it toward the out direction, as illustrated in Figure 3, our design does not restrict the rectification direction but is to encourage a sharp density variation inside of one voxel by both penalizing the front density value meanwhile encouraging the after one. Therefore, in some cases, the Surface Rectification can even help fix unexpected hollows to be smooth and complete (e.g., when the density value of the enter point $\mathbf{P}_e$ has already been small but the out point $\mathbf{P}_o$ is not large enough). The surface would not be blindly pulled towards a certain incorrect direction. 2) As noticed by the reviewer and analyzed in our paper, an accurate surface is always formed by tiny voxels rather than the large ones. As our Surface Rectification is conducted inside specific critical voxels, its direct effect is limited to its target and the nearest neighbors. Thus, for the tiny voxels that are important to form the final accurate surfaces, restrained yet effective rectification is adopted, while the large voxels are still in the stage of coarser optimization with the photometric and geometric supervisions. We'll add more explanations in the revision to better clarify the working principle of Surface Rectification.

• The quantitative and qualitative effect of the Surface Rectification.

Surface Rectification brings a significant and critical improvement in high-precision surface refinement in reconstruction accuracy, especially playing an irreplaceable role in situations where the reconstruction results have already been very close to the ground truths. Different to the proposed contributions like Voxel-Uncertainty Depth that has a large and macro influence with large geometric changes to adjust the obvious errors, Surface Rectification is an important and necessary technique for pursuing the challenging target of extremely high-precision surfaces, which works mainly in the stages of an almost well-reconstructed geometry to certify the delicate and difficult details, such as shown in Figure 7. Especially, its effect can be well demonstrated on the DTU dataset quantitatively, as shown in Table 8 in the supplementary material. Besides the critical accuracy improvement in high-precision cases, we believe the revealed problem behind this technique can also deliver novel insights to inspire more advancements to facilitate the development of the field.

Thanks for the feedback from the reviewer! To help the readers better understand, we'll add clear guidance in the main paper to the related additional experiments in the supplementary material, and provide more qualitative results for better demonstration.

---

Q2: Writing problems.

We sincerely appreciate your careful notice to help improve our paper! In line 230, we have missed a word of "alpha" during writing, and we are sorry for causing the confusion. The correct descriptions should be "... an enter voxel alpha $a_{**p,e**}$ and the out $a_{**p,o**}$ ...", which is calculated following Eq. (9). And following the comment, we have checked our paper thoroughly and corrected the grammatical errors, and fixed unclear expressions that may lead to misunderstanding.

Specifically, for the problems pointed out by the reviewer, we have made the corrections, such as:

• Line 177: "Geometric $U_{\text{geom}}(v)$ is composed of $U_{\text{base}}$ from level with voxel density, ..." -> "Geometric uncertainty $U_{\text{geom}}(v)$ is composed of the level-depended base uncertainty $U_{\text{base}}$ and the voxel density, ...";
• Lines 182-183: "..., we resort to a patch-wise global-local depth loss for better scale alignment and facilitate the geometry knowledge learning." -> "..., we resort to a patch-wise global-local depth loss for better scale alignment and facilitating the geometry knowledge learning.";
• Line 187: "To ensure adaptive and robust for various stages and scenarios, we obtain ..." -> "To ensure adaptive and robust constraints for various stages and scenarios, we obtain ...".

These revisions and the other unlisted writing refinements will be updated in the revised version.

---

Q3: Relation to the initialization from vanilla SVRaster.

Indeed, our method does not inherit reconstruction results from vanilla SVRaster, and is able to perform high-quality reconstruction for the challenging cases that SVRaster may fail. We'd like to note that the pipeline of GeoSVR is not linked to the initial voxels produced by vanilla SVRaster, and is robust in situations where SVRaster can only reconstruct seriously inaccurate surfaces. As described in Section 1 and Figure 3, both our method and SVRaster adopt initialization of constant voxels that depend on the range of camera trajectory to cover the whole scene, and the reconstruction processes would go differently once the optimization starts. As shown in the first scene in the supplementary video, and comparisons of Figure 11 in the supplementary material, GeoSVR has successfully reconstructed the regions that are extremely biased from the ground truth in SVRaster, demonstrating the superior robustness of GeoSVR. Due to the restriction on the rebuttal, we are not allowed to upload more typical cases here with images, but we will add the results in the revised version to help declare the performance relation between our method and SVRaster.

---

Q4: Better clarity of the highlighting and metrics.

Thanks for the suggestions! Following the comments, we have conducted a thorough double-check of our paper, especially on the unclear descriptions, and refined it, including adding the explanations of the color marks in Tables 1, 2, 3, and declaring the directionality of the metrics in Tables 1 and 2 in the captions. The updates will be shown in the next chance of paper uploading. We thank the reviewer again for helping us make the paper easier to read with better clarity.