Factorized Neural Radiance Field with Depth Covariance Function for Dense RGB Mapping

1. The paper's evaluation results lack conviction. There is a notable absence of visualizations for the reconstructed 3D map, 2D novel view synthesis, and pose trajectories. Moreover, there is no 3D metric evaluation of scene reconstruction quality. The solitary 2D metric in Table 1 raises suspicion. The lack of per-scene evaluation results for Orbeez-SLAM and the methodology for averaging results need clarification. The discrepancy in iMAP's PSNR compared to NICE-SLAM, even inferior to the RGB version of NICE-SLAM, requires an explanation.
2. The paper should clarify the procedure used to measure speed. It appears that NICE-SLAM is already slow, but the proposed method is even slower. This conflicts with the claim of being "more efficient than other RGB-D methods." Identifying the bottleneck responsible for this discrepancy is essential.

The overall structure is not polished, and need further revisions. In high level, the abstract is too long and vague, and the method, separated into section 2 and 4, is not easy to follow. The figures miss labels or clear definitions on what is shown. Many of the variables in equations are not defined or defined later in the text. The results are limited, which is only a single page. In detail, the reference citations often times require additional paranthesis, some sentences are incomplete, and many of the procedures are hard to understand. The incompleteness of the writing is the main concen about the paper.

Minor typos:

• Page 2 "on-the-fly continue learning phases" -> on-the-fly continual learning phases
• Page 2 "with no poses provide" -> with no poses provided
• Figure 2 "Color Residule" -> Color Residual
• In Section 3, many of references may need to be put into the paranthesis, as they are not read as part of sentences. There are so many of them, so I cannot list them all here.
• Near the bottom page 4, we need a reference to NICER-SLAM and DIM-SLAM. It seems like they appear in the last paragraph of section 3 in page 5.
• When you write 'i.e. ,', please remove the tailing space.
• In page 5, it is awkward to say "speed up the speed". Please consider rephrasing it.
• An incomplete sentence in page 6: Inspired by recent works on instant reconstruction (Johari et al., 2022; Wang et al., 2023) which leverages Signed Distance Function (SDF) to build the underlying geometry.
• In page 7, third line "condition" -> "conditioned"
• In page 7, "we weights" -> "we weigh"

1. While the theoretical analysis of problem setup seems interesting, it doesn’t seem to provide new insights beyond existing practice in the community. The decomposition into initialisation and on-the-fly optimisation have already been used by previous methods [1].

2. Some equations in theoretical analysis are also confusing. For example: Why Eq (2) is called posterior instead of likelihood? How are $Q$ and $\Sigma$ defined? If Q represents the uncertainty of pose estimation, how can it be used to formulate the quadratic form in Eq (3)? The dimension mismatches.

3. The proposed self-supervised SDF supervision in Eq (7) is very similar to the pseudo-SDF supervision used in previous methods [2, 3, 4, 5]. The only difference seems to be that instead of using measured depth values it uses rendered depth. While this modification seems new, it’s also a bit confusing as the rendered depth value itself also comes from the predicted SDF. There should be more analysis about this formulation, or at least an ablation study is needed to empirically show it works better.

4. In the proposed covariance-guided sampling, why is $\sigma_{n}$ instead of $diag(\Sigma_{l_*})$ used to measure depth uncertainty? Is this a typo?

5. This uncertainty value is also used to dynamically modulate the truncation distance in self-supervised SDF loss. In principle, the truncation distance should be a scene-specific hyper-parameter and shouldn't change its value for each ray. What's the reason for having ray-dependant truncation values? There is also no analysis or ablation studies.

6. The sampling process also seems very complicated, combining uniform stratified, importance and covariance depth-guided, but no ablation study was provided to show it has really improved the result. In the formulation of Eq (9), the depth variance only depends on the sampled pixel location, which means it will not change when increasing sampling stages. Then why do we need to apply the guided sampling in multiple stages?

7. The experiments were not thoroughly done. It only shows comparisons to two RGB-based method, some important baselines, such as [1], [6] are missing. Besides, the proposed method uses a strong pre-trained depth prior, then it is not entirely fair to compare with NICE-SLAM in RGB mode without providing any depth priors. In addition, Ablation studies are missing for every proposed component.

[1] DIM-SLAM: Dense RGB SLAM With Neural Implicit Maps, ICLR'23

[2] Neural RGB-D Surface Reconstruction, CVPR'22

[3] GO-Surf: GO-Surf: Neural Feature Grid Optimization for Fast, High-Fidelity RGB-D Surface Reconstruction, 3DV'22

[4] ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields, CVPR'23

[5] Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM, CVPR'23

[6] DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras, NeurIPS'21

My main concern is that the technical exposition is currently quite poor. The draft needs to be thoroughly revised for better readability. I have listed some typos, comments, suggestions, below but it is not an exhaustive list and the poor readability made it difficult for me to judge the merits of the main idea proposed in the paper. The paper is not ready for publication.

The claim that the proposed method is for online mapping, and suitable for real-time mapping, is incorrect. As described in the paper, there are two stages. It is not clearly stated, but to the best of my understanding, the first stage is where all the image frames are processed and several quantities are learned or estimated, – the neural networks representing the radiance fields, the camera poses and the depth covariance function. Then in the second stage, the images are revisited and depth estimates are refined by leveraging the radiance field learned in the first stage, as well as the camera poses and the depth covariance function, etc. (based on the technical overview in Section 2 and the graphics presented in Figure 2). The second stage quite efficient and has low running time (5 Hz). However, it is incorrect to call such a system an online SLAM or mapping system as the whole system including the stage that computes poses is not running in real-time.

After reading Section 4.1 and 4.2 carefully, I was unable to understand how the camera poses are computed either during the initialization stage or afterwards, i.e. whether they are further refined during the on-the-fly stage. Various losses are defined in the various equations, but it is not completely clear how they are used in the algorithm.

The evaluation is based on the L1 depth error between the estimation and GT depth map, but to compute the relative scale, the median depth values in the estimated and GT depth maps are used. This metric is not appropriate as it does not penalize scale drift in the reconstruction. Instead, a more appropriate approach is to compute a single relative scale factor for the complete scene reconstruction, and using that in the metrics computation.

-------- comments, typos, etc. -------------

Section 4.1: “we denote factorized neural radiance field as 4D representation G … “ --> Earlier in the paper, the term G is used to denote the NeRF function when describing prior work, but the mathematical form was not precisely defined. It is unclear whether the function G is the same as one that was used in previous papers, or different. Nor is the factorization of G actually explained anywhere. Considering that “Factorized Neural Radiance Field” is in the paper title and one of the core ideas behind the proposed approach, it must be properly defined and explained.

Typos:

page “scarify their real-time ...” -> sacrifice their real-time ..,? not sure. “we tackle to” -> incorrect grammar “on-the-fly continue “ -> continuous,

“where the prior aims to learn mapping without poses” – what does this phrase mean? Please rephrase and explain clearly.

“later to” should be “latter to”

“To maintain the function’s stability”  does function refer to the kernel function? The term “stability” needs a proper definition and explanation.

During on-the-fly phase, our method achieves real-time high-fidelity mapping  when does the initialization phase end and the on-the-fly phase start?

page3: “warpping loss” “We formulate .. “  we formulate …

Section 2: Problem Setup “… In this setup, given image frames I and corresponding pose \tilde{P}, we estimate the frames’ color \tilde{I} and depth \tilde{D} to reconstruct dense RGB map \tilde{m}. But it is unclear what \tilde{m} actually is? It needs to be defined. This makes it difficult to understand what Equation (2) actually means. I can try to guess, but my concern is that the reader should not have to guess these things. Similarly, I have a hard time understanding what the functions G and \Phi (decoder) take as input and return as output.

page 4: Despite of -> Despite The following sentence is unclear: "Therefore, it's crucial to maintain a relatively accurate depth to guide covariance function to infer depths without self-supervised training described in Sec 4.1." -- the sentence is confusing because it talks about self-supervised training and inference at the same time. Also, it is unclear how the goal to "maintain accurate depth" is actually achieved.

page 5: “To cultivate the potential of representation ability of .. “ -- sentence is unclear. Must be rephrased.

page 6: “… calculate the difference inside the visibility mask B.” -- what is B? it needs to be defined. Where does B come from, i.e. how is it computed? “denotes the penetrating light”  this phrase should be rephrased.