Optimize the Unseen - Fast NeRF Cleanup with Free Space Prior

We thank the reviewer for the feedback, we encourage the reviewer to view the HTML visualization video in the supplementary as one of the reviewers mentioned it as a helpful tool to understand the good results. Below are our clarifications and answers:

SSIM and LPIPS Performance

We appreciate the reviewer's attention to the SSIM and LPIPS metrics. We understand the confusion may arise from comparing methods at different coverage levels. We would like to clarify that our method achieves comparable performance to baseline methods when compared at equivalent coverage levels. We will enhance this clarification in the paper to make the comparison clearer.

Our method achieves virtually identical SSIM and LPIPS performance to BayesRays at equivalent coverage levels, while being 2.5× faster in inference.

When comparing at equivalent coverage levels (refer to Table 2 in supplemental - Full Results):

Coverage ~92%-93%:

Coverage ~89%-90%:

Uniform Sampling Effectiveness

We respectfully disagree with the assessment that uniform sampling is overly aggressive. Our results demonstrate that we achieve state-of-the-art performance in PSNR, SSIM, and LPIPS while maintaining high coverage. Both qualitative and quantitative results support the effectiveness of our approach without evidence of over-smoothing or detail loss.

The Regularization Term (Choosing $\\lambda$)

We would like to address the concern about $\\lambda$ tuning. Our method demonstrates robustness across hyperparameter choices rather than requiring careful tuning. All visualizations in the paper were generated with $\\lambda=0.1$ across different NeRF methods and scene characteristics, demonstrating universal applicability.

Our extensive ablation study (Appendix A.5) shows that our method's mechanism is more important than precise $\\lambda$ tuning:

"Our method is robust to a wide range of $\\lambda$ values as can be seen in Table 2, $\\lambda \\in [10^{-5}, 1]$ yields both high PSNR (17.35–17.69) and high coverage (87.57%–96.45%). This robustness stems from our method's design, which optimizes regions where the reconstruction loss is inactive."

Our "Balancing Cleanup and Scene Integrity" section (line 157) explains the 3 regions of interaction between the losses:

• Unseen regions: Only FSP loss acts, pushing densities to zero
• Empty seen regions: Both losses agree on zero density
• Surface regions: NeRF reconstruction rays provide strong, consistent supervision by repeatedly sampling the same locations, while the sparse FSP loss only occasionally samples these areas and therefore cannot overpower the reconstruction signal

You can observe Figure 4 to see how density in empty regions is cleaned while preserving surface structure.

Trade-off Analysis

While BayesRays offers a wider PSNR-Coverage range, this can be misleading. As shown in Figure 5 and Figure 3 and discussed in lines 250-253:

"Striking a balance between PSNR and coverage is crucial, as high PSNR with low coverage can be misleading. For instance, as shown in Fig. 5, BayesRays at a 0.3 threshold achieves the highest PSNR but suffers from low coverage, leading to noticeable gaps in the reconstructed scene (Fig. 3) that compromise overall image quality."

Our method maintains both high PSNR and coverage in the practical operating range.

Responses to Specific Questions

Q: What causes the Free-Space Prior to yield worse SSIM and LPIPS values, indicating lower performance on perceptual metrics?

A: As demonstrated above, our method does not yield worse values when compared at equivalent coverage levels.

Q: How does the Free-Space Prior affect fine structure reconstruction? Does it impact surface reconstruction accuracy?

A: Our results indicate no negative impact on fine structure reconstruction. Figure 4 demonstrates that wall density is well-preserved, and surface reconstruction remains accurate. Both qualitative and quantitative results support maintained reconstruction quality.

Q: What is the best approach for selecting hyperparameters across different scenes?

A: We recommend using $\\lambda=0.1$ universally. Our robustness analysis (Appendix A.5) shows this works well across diverse scenes, with the method's core mechanism being more important than precise tuning.

I agree with reviewer pWFp that the degradation of fine geometric details might be a concern, and it would be cool to asses this quantitavely. Using Chamfer Distance on point clouds might, thought, be misleading, since there is no guarantee that the starting point cloud is correct, especially given the presence of floaters and other artifacts. I'm also not aware, given my lack of experise on the topic, of whether Chamfer Distance can be used to evaluate surfaces. In general, checking geometry in 3D in a quantitative way is difficult without a proper ground truth, particularly due to the poor representation of the initial NeRF geometry before any cleanup. There might be a significant difference just because artifacts were removed.

A possible experiment could be to compare the PSNR between the training rendered images before and after the cleanup. Alternatively, it might be useful to check whether the surface normals or depth maps remain consistent before and after the cleanup, at least on the training images.

Please correct me if I'm saying something wrong, or if this doesn't make sense in light of what the authors and reviewers think.

Thank authors again for the extended experiments. The dense-view experiment shows that the performance of free-space loss may slightly influence the performance, but marginally. Overall, I think authors demonstrate the free-space loss will not significantly degrade of original quality. I appreciate the simplicity of this method and will raise my score accordingly.

We thank the reviewer for the positive assessment and particularly appreciate the recognition of our visualization and experimental quality. Below are our clarifications and answers:

Addressing Novelty Concerns

While our core contribution is indeed a simple post-hoc loss function, we believe this simplicity is a strength rather than a limitation. The method's elegance lies in achieving SOTA results through a principled, theoretically grounded approach that generalizes across diverse NeRF architectures and datasets without requiring architectural modifications or learned priors.

Selective Point Treatment

The reviewer's suggestion about reasoning over "correct" points is insightful and represents an interesting direction for future work. While our current approach takes a simpler strategy, our "Balancing Cleanup and Scene Integrity" mechanism (line 157) naturally achieves similar outcomes through the interaction between losses. This creates three distinct regions of interaction between the losses:

1. Unseen regions: Only FSP loss acts, pushing densities to zero
2. Empty seen regions: Both losses agree on zero density
3. Surface regions: NeRF reconstruction rays provide strong, consistent supervision by repeatedly sampling the same locations, while the sparse FSP loss only occasionally samples these areas and therefore cannot overpower the reconstruction signal

You can observe Figure 4 to see how density in empty regions is cleaned while preserving surface structure through this natural partitioning that allows our method to preserve well-reconstructed regions while cleaning artifacts.

We thank the reviewer for the feedback, we encourage the reviewer to view the HTML visualization video in the supplementary as one of the reviewers mentioned it as a helpful tool to understand the good results. Below are our clarifications and answers:

Training Time Clarification

The training time refers to the "cleanup phase" and we will clarify this in the paper. Each method uses a fundamentally different training approach: our method performs fine-tuning using the Free Space Prior and Reconstruction Losses, Nerfbusters fine-tunes with a large pre-trained 3D diffusion model, while BayesRays collects Hessian data for uncertainty estimation. The details about each method's distinct training phase are in lines 274-277:

"For training time, Nerfbusters employs a pre-trained 3D diffusion model to fine-tune NeRF per scene, resulting in a high training time of approximately 20 minutes per scene. BayesRays is faster, taking 37 seconds to gather data for Hessian computation, while our approach is the fastest, completing cleanup training in just 25 seconds."

As for the number of iterations and samples, it is stated in the implementation details (Lines 455-456), we will clarify that in the runtime section too:

"In our method, density optimization was achieved with 1,000 iterations using $2^{17}$ randomly sampled points across the 3D scene space."

Inference Speed Advantage

The reviewer asked how we achieve faster inference without optimizations to the inference process. The key difference lies in the inference architecture: BayesRays requires additional uncertainty thresholding computations during rendering, resulting in 506ms per frame, while our method uses only the fine-tuned NeRF without any additional overhead, achieving 205ms per frame on the same hardware - 2.5× faster than BayesRays, which is the only SOTA method achieving comparable cleanup quality.

Hyperparameter $\\lambda$ Robustness

We understand your concern regarding different scenes with different numbers of images. We claim that the method mechanism is more important than the specific $\\lambda$ parameter. To clarify, all visualizations in the paper were generated with $\\lambda=0.1$ across different NeRF methods and scene characteristics, as mentioned in line 178.

This robustness can be explained by two key mechanisms:

1. Fixed sampling strategy: The number of samples and number of rays are fixed in each iteration, regardless of the number of images in the scene. This ensures consistent optimization behavior across different scene configurations.

2. Balanced loss interaction: Our "Balancing Cleanup and Scene Integrity" section explains the 3 regions of interaction between the losses:

   • Unseen regions: Only FSP loss acts, pushing densities to zero
   • Empty seen regions: Both losses agree on zero density
   • Surface regions: NeRF reconstruction rays provide strong, consistent supervision by repeatedly sampling the same locations, while the sparse FSP loss only occasionally samples these areas and therefore cannot overpower the reconstruction signal

You can observe Figure 4 to see how density in empty regions is cleaned while preserving surface structure through this balanced interaction.

Moreover, our ablation study demonstrates robust performance across a very large range of $\\lambda$ values (Appendix A.5 - Choosing $\\lambda$):

"Our method is robust to a wide range of $\\lambda$ values as can be seen in Table 2, $\\lambda \\in [10^{-5}, 1]$ yields both high PSNR (17.35–17.69) and high coverage (87.57%–96.45%). This robustness stems from our method's design, which optimizes regions where the reconstruction loss is inactive."

The Organization of the Experimental Section

Thank you for this feedback, which helps us improve the paper. We will move the ablation study of hyperparameter $\\lambda$ and the comparison between fine-tuning and training from scratch to the main text.

We thank the reviewer for the feedback, we encourage the reviewer to view the HTML visualization video in the supplementary as one of the reviewers mentioned it as a helpful tool to understand the good results. Below are our clarifications and answers:

"Nothing in Unseen Regions" Assumption

This assumption is not strong, but rather fundamental to NeRF reconstruction. If training images do not cover an area, it cannot be meaningfully reconstructed - this is a core limitation of photometric optimization. Our contribution is in enforcing zero density in these regions instead of allowing arbitrary noise accumulation.

Regarding the concern about "non-object centric scenes", we would like to highlight that our experimental evidence demonstrates effectiveness across many non-object-centric scenes throughout our paper. Notably, the Nerfbusters dataset itself is not object-centric. Our method performs well on complex indoor and outdoor scenes with multiple objects, backgrounds, and varied scene compositions, which we believe addresses the concern about applicability to non-object-centric scenarios.

Additionally, our approach generalizes to many NeRF variants and scenarios such as underwater, semantic, and language-embedded NeRFs as shown in Figure 1, which distinguishes our method from other state-of-the-art cleanup approaches.

The Regularization Term

We believe there may be a misreading of Figure 5: increasing $\\lambda$ actually improves PSNR (while decreasing coverage as expected). This relationship can be verified in Table 2 of the supplementary material.

It is true that the regularization term interacts with scene rays, but we discuss this extensively in the paper. This is not a limitation, but the core mechanism of our method.

Our "Balancing Cleanup and Scene Integrity" section (line 157) explains the 3 regions of interaction between the losses:

• Unseen regions: Only FSP loss acts, pushing densities to zero
• Empty seen regions: Both losses agree on zero density
• Surface regions: NeRF reconstruction rays provide strong, consistent supervision by repeatedly sampling the same locations, while the sparse FSP loss only occasionally samples these areas and therefore cannot overpower the reconstruction signal

These are illustrated in Figure 4, which shows how density in empty regions is cleaned while preserving surface structure.

Universal $\\lambda$ Parameter

$\\lambda$ is not hand-chosen for different scenes as mentioned in line 178. All visualizations in the paper were generated with $\\lambda=0.1$ across different NeRF methods and scene characteristics, demonstrating universal applicability.

Our extensive ablation study (Appendix A.5) shows that our method's mechanism is more important than precise $\\lambda$ tuning:

"Our method is robust to a wide range of $\\lambda$ values as can be seen in Table 2, $\\lambda \\in [10^{-5}, 1]$ yields both high PSNR (17.35–17.69) and high coverage (87.57%–96.45%). This robustness stems from our method's design, which optimizes regions where the reconstruction loss is inactive."