ProvNeRF: Modeling per Point Provenance in NeRFs as a Stochastic Field

Thank you for finding our theoretical extension of IMLE to stochastic processes novel and mathematically rigorous and demonstrating improvements over existing methods. We address the questions raised below.

Q1 Introduction still leaves me asking “Why is this an important problem?”

Modeling provenances enables analysis of NeRF reconstruction from a classical stereo-matching point of view. As noted by the other reviewers, provenances are important concepts to model because they shed “insight about the quality of the input data for reconstruction” (GM1F) that “leads to a combination of traditional 3D vision-related fields, e.g. triangulation and uncertainty, to novel view synthesis” (u5T3). We will modify the introduction for better clarity and emphasis on the importance of provenance in the final version.

Q2 Method only addresses triangulation uncertainty.

Yes, one application of modeling provenances is modeling triangulation uncertainty in NeRFs. While uncertainty within NeRF’s reconstruction is multifaceted, isolating triangulation uncertainty can benefit downstream tasks such as next-view augmentation [19] that concern the capturing process (note other uncertainties such as transient are intrinsic to the scene). In the future, we hope to combine other types of uncertainty with triangulation uncertainty to achieve a more comprehensive understanding of NeRF reconstruction.

Q3 Mathematical presentation; should clearly state the use case up-front.

Thank you for your valuable feedback; we will modify our paper accordingly in our final version. Our ProvNeRF is a NeRF training add-on and we will state that up-front. We intend to describe the method in a mathematically precise way, but instead we will in the final version lighten the mathematical notations, simplify the idea's presentation and move details to the supplementary.

Q4 Stochastic process in the exposition of Sec 4.1.

Our stochastic process is defined as a collection of distributions at each 3D location instead of distribution over different 3D locations. To provide an illustration, Figure 2(a) in our rebuttal shows the provenance direction samples at different 3D locations. We will include this in the final version and add further clarifications to avoid confusion.

Q5 Clarification on visibility.

By visibility, we refer to that given by the training dataset (so the former in your question), not from the scene. We will clarify this in our final version. But modeling visibility of the scene would be an interesting follow-up work.

Q6 Stochastic process v.s. Stochastic field.

Yes, stochastic field would be a more suitable name for our provenance model, as each of our samples is a function mapping each 3D location to a provenance of it. We note that there is an inconsistency in the stochastic process literature in terms of terminology. For example, [74] defines stochastic process as an n-dimensional and m-variate random function whereas [75] defines it as those with only a one-dimensional indexing set. However, stochastic field is a more suitable name given that our indexing set is a field. We will change the stochastic process to the stochastic field in our final version. Note that we still use the term provenance stochastic process in rebuttal for terminology continuity.

Q7 Why are directions not unit length?

The predicted direction samples are not unit length to handle object/scene occlusions. We model a visibility term $v \in [0,1]$ (c.f., Eq. (5), main) as the norm of the predicted direction that accounts for how occluded this provenance sample is: this is why the distance-direction tuple lives in $\mathbb{R}_+ \times \mathbb{D}^3$. We overloaded the notation and used $\bf d$ for both the unnormalized and normalized directions. In practice, once we extract a provenance’s visibility by computing the norm of the direction, we normalize the direction and use it for downstream tasks. In the final version, we will denote the normalized direction as $\tilde{\bf d}$ and fix the inconsistencies in the paper. For further clarity, we note that our visualizations show the negative of the provenance direction (for clearer visuals) – that is the direction from a 3D location should be pointing away from the camera.

Q8 $o_i$; defining Eq. (5) before (4).

Yes, $o_i$ is the camera origin (see Ln. 99 main). We will reorder Eq. (4) and (5) in the final version; thank you for your suggestion.

Q9 Architecture of $H_\theta$.

It is a 3-layer MLP with ReLU non-linearity and input feature, hidden feature, and output dimensions being 288, 256, and 4, respectively. We will include these details in Sec. S2 in the final version.

Q10 Limitation.

Thanks for the suggestion. We will move our limitation discussion to the main paper.

Q11 Longer discussion on method’s applicability.

Our method can be applied to model stochastic fields other than provenance fields. For example, an interesting application other than provenance modeling is to model the material properties of a NeRF as a stochastic field. For instance, we could model the BRDF at each point in 3D as a stochastic field over the incoming and outgoing solid angles. Learning such a stochastic field would allow us to enable interesting applications such as relighting or material modification. Another possibility is to extend the deterministic framework in [76] to model the equivalent classes of signed distance fields given by a set of discrete signed distance samples as a stochastic field. This allows us to sample different but all plausible signed distance fields (thus implicit surfaces) given a set of discrete signed distance samples. We will include this discussion in the final version.

[74] Shinozuka, M. (1987). Stochastic Fields and their Digital Simulation.

[75] Knill, O. (2009) Probability Theory and Stochastic Processes with Applications.

[76] Sellán, S., et. al. (2024). Reach For the Arcs, SIGGRAPH 24’

Thank you for finding our work interesting and insightful and our evaluations thorough and compelling. We address the questions raised below.

Q1 Presentation of the method section.

Thank you for your valuable feedback. We will incorporate your suggestions and promise to improve the presentation of the method in the final version. We originally intended to describe the method in a mathematically precise way, but we understand this can cause confusion. We will lighten the mathematical notations, simplify the presentation of the idea, and move details to the supplementary. We will describe the method in a more intuitive way in the final version.

Q2 Clarification on length and norm of direction vector.

The length of the predicted direction determines how visible the provenance direction is to handle object/scene occlusion. That is, we model a visibility term $v \in [0,1]$ (c.f., Eq. (5) of main) as the norm of the predicted direction that accounts for how “visible” this provenance sample is. For further clarity, we note that we overloaded the notation and used $\bf{d}$ for both the unnormalized and normalized directions. In practice, once we extract a provenance’s visibility by computing the norm of the direction, we normalize the direction and use it for downstream tasks. In the final version, we will denote the normalized direction as $\tilde{\bf d}$ and fix the inconsistencies in the paper. Finally, in our NVS regularizer formulation in Eq. (10), we use the length of the direction vector to distinguish floaters from solid surfaces. I.e., if the predicted direction has $\lVert\bf d\lVert<0.7$, we assume that there is a solid surface in that direction and it shouldn't be regularized. Otherwise if $\lVert\bf d\lVert\geq0.7$, we treat it as a floater where we apply our regularizer to remove it. We will add this clarification in the final version.

Q3 Variation in length of direction vectors.

They are varied, as shown by the length of vectors in the supplementary video. We further provide additional visualization in Figure 2 (a) in the rebuttal PDF; we will include this in the final version. We note that the visualization in Figure 6 only shows normalized directions. We apologize for this confusion and we will change it to reflect the length of the direction vector in the final version.

Q4 Misalignment of provenance visualization.

Yes, you are right. The visualizations in Figure 6 and the video show the negative of the predicted provenance directions. This is done for ease of illustration: otherwise the samples will be hidden by the scene. We will add this clarification in the final version.

Q5 Ln. 128.

Thanks for pointing this out. We will fix it in the final version.

Q6 Clarification of the provenance visualization.

Yes, the visualizations shown in the video and Figure 6 are provenance samples generated at the ground truth geometry surface provided by the dataset. We will include this clarification in the final version.

Q7 Ln 168.

Thanks for the suggestion. Yes, the expression in Eq.(6) is a more mathematically precise definition of our random function $\bf{Z}$. We will modify the wording for a more precise definition in the final version.

Thank you for finding our method interesting, easy to follow, and enriches NeRFs with critical insights on triangulation quality.

We here clarify our experiment setup and avoid confusion. Our main experiments are split into two parts, each corresponding to a different application using ProvNeRF. 1) Novel view synthesis (Sec. 5.1): we use our learned provenance to formulate an additional regularizer $\mathcal{L}\_{provNVS}$ to improve scene reconstruction. We evaluate our approach in Fig. 3 and Tab. 1, demonstrating improvements compared to baselines. 2) Triangulation uncertainty estimation (Sec. 5.2): we use our learned provenance to estimate triangulation uncertainty. We show results on uncertainty estimation in Fig. 4 and Tab. 3 against baselines by measuring the uncertainty’s correlation to depth errors. This is a common evaluation protocol in prior works [51-52]. Note that in this context we do not compare the depth error across different models. We address your questions below.

Q1 Evaluation of the proposed idea.

As highlighted by other reviewers, we provide a “thorough” (GM1F) evaluation of our method and achieve improvements in both reconstruction and triangulation uncertainty metrics(o98v). In the attached PDF, we provide additional experimental results, including applying ProvNeRF on 3DGS for uncertainty estimation (Figure 1) and a depth map comparison between our method and SCADE (Fig. 2(b)).

Q2 “Modeling the provenance improves NeRF's accuracy”.

We clarify that modeling the provenance alone does not improve the NeRF’s reconstruction. However, it can be used to formulate the $\mathcal{L}\_{provNVS}$ regularizer to improve NVS and scene reconstruction as shown by experiments in Sec. 5.1. We further provide improved depth rendering visuals in Figure 2(b) of the attached PDF that supplements Fig. 3.

Q3 “Few direct comparative experiments on depth/position, with only Fig. 4 showing some reflection.”

We clarify that the depth error maps in Figure 4 are NOT to show improvement in depth as it is part of our uncertainty experiments (Sec. 5.2). Note that our $\mathcal{L}\_{provNVS}$ regularizer is not used here. Instead, Fig. 4 visualizes uncertainty maps from different methods and compares their correlations to the depth errors, which serve as a qualitative comparison for uncertainty estimation.

Q4 Tables 1 and 2: “I wouldn’t say results significantly improved”.

As noted by GM1F, while the improvements in the quantitative metrics on the NVS experiments (Table 1 & 2) are not huge, “several examples where some of the common artifacts present in indoor NeRFs are mitigated if not completely removed.” making our method “quite compelling”. This suggests that our additional regularizer $\mathcal{L}\_{provNVS}$ can improve the NeRF reconstruction by modeling provenance that comes for free from the training images.

Q5 Difference between Bayes’ Rays in Figure 4.

Sec. 5.2, Fig. 4 compares our approach against baselines such as Bayes’ Rays on uncertainty estimation. Note that a good uncertainty map should correlate well with depth error maps, marking regions with high depth errors as highly uncertain and vice versa. The boxed regions in Fig. 4 demonstrate the superiority of our estimated uncertainty over Bayes’ Rays. For example, our method accurately identifies both chairs (left and right examples) as certain due to good triangulation, contrasting with baselines that incorrectly mark them as uncertain despite their low depth errors. Again, we clarify that the results in Fig. 4 (main) do not show improvement in scene reconstruction – we do not enforce our $\mathcal{L}\_{provNVS}$ regularizer here. In fact, the depth error for Bayes’ Rays and ours are the same in Fig. 4 as we use the same base model to compute for uncertainty. For improvements in scene reconstruction see our NVS experiments (Sec 5.1.) where we enforce the regularizer to clear out floaters.

Q6 Figure 2 caption.

Thanks for pointing this out. We will fix this in our final version.

Q7 Does the method solve the uncertainty problem and why is it important?

Yes, our proposed method can be used to estimate triangulation uncertainty as shown in Sec. 5.2. Uncertainty estimation is crucial for applications needing reliable reconstruction like continual learning and robotic navigation. Several studies [51-52, 55] have addressed this within NeRF settings.

Q8 Results are not SotA.

We achieve SotA results in both NVS and uncertainty estimation applications. In the NVS experiment, Figure 3 shows our results clearly improve the previous SotA method SCADE by removing floaters and fuzziness on surfaces. This is also reflected quantitatively in Table 1 as we achieve on average better than the baselines on both datasets and ours are better in almost all the metrics; In the uncertainty experiment, qualitative and quantitative comparisons suggest that our uncertainty map is more informative -- i.e., marking poorly triangulated regions with high uncertainty and vice versa -- compared with existing baselines.

Q9 Applying ProvNeRF to 3DGS.

Yes, we can apply our method to 3DGS. We apply ProvNeRF on top of 3DGS on Scannet dataset. The post-optimization takes around 30 minutes per scene on a single NVIDIA A6000 GPU. To study its applicability, we used our trained provenance stochastic process to estimate triangulation uncertainty as in Sec. 5.2. See our global response for implementation details. Figure 1 in the attached PDF shows the comparison of estimated uncertainty maps between ours and FisherRF [73]. Note that we obtain a more correlated uncertainty map w.r.t. the estimated depth error from 3DGS. The same figure also shows a NLL comparison against FisherRF where we outperform the baseline by a large margin. These results show that applying ProvNeRF to recent explicit representations such as 3DGS is promising and we leave further exploration as future works.

[73] Jiang, W. et. al. (2023). FisherRF, CVPR 24’

We thank the reviewer for reading our rebuttal and eliciting their concerns which we address below.

Weird to connect uncertainty directly with depth error

We follow prior works [19, 51-52, 73] to visually evaluate our uncertainty estimation by inspecting its correlation with the depth error in Figure 4 of the main. We didn’t put the rendered depth maps in the figure because we are a post-hoc uncertainty estimation method, i.e. our approach is a plug-in to any NeRF backbone to estimate triangulation uncertainty, which means that the depth maps we obtain will be exactly the same as the backbone. However, baselines like CF-NeRF [51] and Stochastic NeRF [52] are not post-hoc methods and need to be trained from scratch with a modified volumetric rendering pipeline. So their convergence is not guaranteed and in fact, produces blurrier results than SCADE, the backbone we use. Our work is more similar to the recent baseline Bayes’ Rays [19], which also only shows a comparison between the uncertainty map and the depth error map and does not show rendered images or depths of the backbone (c.f. Fig. 6 of [19]).

Comparison with SCADE; the USA flag in the second column in pdf. Fig.3.

The error in the USA flag in Fig. 3 of the main is due to a pose inaccuracy on the dataset released by SCADE, a common issue from using COLMAP to estimate camera poses. This causes inconsistencies in the optimization, resulting in the wrong geometry of the flag. In fact, all of the methods in Table 1 take these poses as input and they all have this problem as well. This can be mitigated if we obtain better camera poses for the dataset. However, it is important to note that this problem is orthogonal to our contribution in Sec. 5.1 where we use our provenance stochastic process to remove artificial floaters, which is a common artifact in indoor NeRFs as suggested by GM1F and is mitigated by our method as shown in Fig. 3.

It seems the proposed method is good at filling the blank, not improve the reconstruction quality.

Our method does not fill in the blank. We are an optimization-based method and do not use any generative priors. Instead, the improvements in Fig. 3 are direct results of our method removing floaters from the pretrained SCADE model and revealing the correct geometry behind them. This is also suggested by the depth maps renderings we show in Fig 2 (b) of the rebuttal PDF.

We hope we have answered your questions and concerns so that they can assist your final decision. Please do not hesitate to ask us should there be anything unclear.

Thank you for finding our method sound, theoretically complete, and has the potential to inspire follow-up works in the field. We answer the questions raised below.

Q1 Alternatives in modeling the probability distribution.

Instead of as samples, we can either represent provenances as a discrete distribution in Eq.(4-5) main or as a closed-form continuous distribution (e.g. Gaussian Mixtures). However, discrete representations incur discretization errors and closed-form distributions have limited expressivity. In contrast, our provenances use an expressive deep net $H_\theta$. See Table S1 for an NVS quality comparison.

Q2 Alternatives to functional IMLE.

Compared to other sample-based implicit probabilistic models like GAN and VAE which are prone to mode collapses, IMLE can model multimodal distributions by implicitly maximizing likelihoods. Table 4 and Table S1 show quantitative comparisons of fIMLE against VAE highlighting its superior performance. We also tried GAN but it didn’t converge.

Q3 Visual results and caption improvement.

We include additional visualizations in the rebuttal. In Fig. 2 (a), we show provenances sampled on the ground truth surface. In Fig. 2 (b), we show improvements to depth images after applying $\mathcal{L}\_{provNVS}$ to SCADE. We will include them in the final version and update captions for Fig. 3, 4, and 6 of main for clarity.

Q4 Uncertainty’s connection with depth error.

A poorly triangulated region likely has incorrect depth due to large depth ambiguity (c.f. Fig. 5). Consequently, regions with high depth error should have high triangulation uncertainty and vice versa. Thus, we validate our uncertainty map by examining its correlation with depth errors, which is a common approach in prior works [51-52].

Q5 What are considered improvements in uncertainty maps?

A good uncertainty map should correlate well with depth errors. E.g., the boxed regions in Fig. 4 (main) demonstrate the superiority of our uncertainty over the baselines. Our method accurately identifies both chairs as certain due to low depth errors while baselines incorrectly mark them as uncertain.

Q6 Floaters/Fuzziness not addressed totally.

Fig. 3 (main) shows visuals from the NVS experiment using our $\mathcal{L}\_{provNVS}$, improving scene reconstruction by clearing most floaters. Rendered depths included in Fig. 2 (b) of the rebuttal show further validation. While our method in theory may not eliminate all the artifacts when they are invisible from training cameras, in practice it removes most floaters as shown in the figures above. In Fig. 4 (main) that compares uncertainty estimation methods, floaters are present because we didn't apply the $\mathcal{L}\_{provNVS}$ regularizer; these floaters are from the pretrained SCADE.

Q7 Uncertainty evaluation.

NLL, a common uncertainty metric [51-52, 55], measures the negative log-likelihood of ground truth surfaces under the model's uncertainty prediction. The metric is intuitive as effective uncertainty maps should assign high likelihood to the true scene surface. While Area Under Sparsification Error (AUSE) is also used [19. 73], Figure S5 shows its defects that can lead to unreliable scores. We therefore opted to evaluate NLL.

Q8 Provenance parameterization.

Yes, we can model provenance as a 3D vector that connects the observation center with the 3D location. Other parameterizations such as a camera pose in se(3) are also viable.

Q9 3DGS discussion.

Our method is compatible with 3DGS. We integrate ProvNeRF to 3DGS and assess its triangulation uncertainty using the method in Sec. 5.2. See Fig. 1 of the attached PDF for both visual and quantitative comparisons with FishRF [73]. The results demonstrate the potential of applying ProvNeRF to explicit representations like 3DGS, and we plan to explore this further in the future. Details are in our global response.

Q10 Experiment setting: should be sparse and why use scannet, tanks and template dataset.

Our experiments indeed use a sparse setting following SCADE [58]. Specifically, ScanNet and T&T are standard datasets [58] with 18-26 training images in unconstrained camera poses in each scene. We also test NVS on the In-the-Wild dataset from [58] achieving superior metrics over SCADE:

Q11 Fig. 2 clarification.

Fig. 2 illustrates that modeling provenance is not straightforward, as different triangulation-related phenomena [20] such as camera baseline distance and occlusions need to be taken into account. We’ll improve this figure and caption in the final version.

Q12 Provenance visualization. We provide additional visualization of our learned provenances in Fig. 2(a) in the attached PDF. Note that we visualized provenances together with the training cameras in the supplementary video.

Q13 “Ours” built on NeRF?

Ours is built on SCADE [58] (c.f. Ln 232-233). The only difference from SCADE is the extra $\mathcal{L}\_{fIMLE}$ loss in Eq. (8).

Q14 How sampling is done in practice.

D is sampled every 1000 iterations following IMLE [32], and (x, t, d) are sampled every iteration.

Q15 $\mathcal{L}\_{provNeRF}$ v.s. $\mathcal{L}\_{provNVS}$.

$\mathcal{L}\_{provNeRF}$ is crucial for learning the provenance stochastic process while $\mathcal{L}\_{provNVS}$ leverages this process to enhance scene reconstruction. Ablation studies show that training without $\mathcal{L}\_{provNeRF}$ leads to slightly worse results (Ours* in Table S1). This is because joint optimization synergistically adapts the provenance samples to the current geometry during joint optimization. Conversely, omitting $\mathcal{L}\_{provNVS}$ results in 21.44, 0.716 and 0.349 for PSNR, SSIM and LPIPS respectively. This is inferior to the pretrained SCADE because $\mathcal{L}\_{provNeRF}$ does not improve the reconstruction using provenances.

[73] Jiang, W. et. al. (2023). FisherRF, CVPR 24’