UC-NERF: Neural Radiance Field for Under-Calibrated Multi-View Cameras in Autonomous Driving

1) The writing of introduction.

Thanks for your suggestions. We have revised the introduction to better understand the multi-camera system and the motivation for virtual warping.

2) The usage of MVS methods.

We use the public model of CER-MVS without finetuning. Virtual warping can enhance color correction and rendering quality, regardless of the depth sparsity. We also demonstrate its effectiveness with sparse lidar input in Tab. 1 below.

Table 1. The performance of rendering with different sources of depth for virtual warping. Tested on a randomly selected Waymo scene.

3) The explanation of $**b**$ and $**d**$ in Eq.5.

Thanks for your suggestion. In Eq. 5, $**b**$ and $**d**$ should be $**x**$ and $**y**$ respectively, which represent the translation term of the color correction matrix. We have revised them in the paper.

4) Whether the relative transformation is optimized?

Yes. The relative transformation is optimized.

5) The final training loss is missing.

Thanks for your suggestion. We add the final training loss in Sec. 3.5 of the revised paper.

6) The pose refinement step relies on key points, which could introduce outliers.

We have included the RANSAC[1] algorithm for filtering outliers before bundle adjustment. Generally, pose optimization is based on point correspondences or photometric consistency. However, images taken by different cameras have significant color differences, which violates the assumption of photometric consistency. Furthermore, joint optimization of NeRF, color correction, and poses based on photometric loss only might not provide sufficient constraints.

In contrast, urban scenes are typically endowed with rich textures, which allow for the extraction of a substantial number of key points. We posit that the establishment of image correspondences serves as a more effective and direct strategy for pose optimization. We demonstrate that optimizing poses through explicit pixel correspondences is better than NeRF-pose joint optimization with photometric loss in the multi-camera setting in Tab. 6 and Fig. 12 of our revised paper (in Appendix A.3.2).

7) The consideration of DBARF[2] and FlowCam[3].

Thanks for your suggestion. We finetune FlowCam and DBARF to our Waymo scenes, with results in Tab. 2 below. The performance of our method is better than FlowCam and DBARF in a multi-camera setting. DBARF optimizes poses through cost feature maps between images, which can be affected by significant color differences in images of different cameras. FlowCam requires the calculation of optical flow for pose optimization. However, given the restrained overlap between images from different cameras, the effectiveness of current optical flow techniques and subsequent pose estimation is limited in a multi-camera setting.

Table 2. Comparisons of our method with FlowCam and DBARF. Tested on a randomly selected Waymo scene.

8) The change of the relative camera poses.

Thanks for your suggestion. As you pointed out, the relative pose between cameras may change during driving, but such considerable changes are typically caused by accumulated shakes over a vehicle's long-term drive like months. NeRF models generally operate on short video sequences lasting from minutes to hours, during which the changes in the relative poses between cameras are subtle. On the contrary, directly modeling the transformation between cameras without our proposed spatiotemporal constraint is feasible, but it may lead to less accurate pose estimation. This is primarily because of the fact that images captured by a moving vehicle share limited overlaps, especially among the images captured by different cameras. This could potentially result in insufficient connections between images taken by different cameras, making the pose optimization relatively less constrained and unstable. Considering the two factors, we propose to utilize the spatiotemporal constraint to enhance the pose refinement.

We add an experiment in Tab. 3 below to analyze the spatiotemporal constraint. The experiment is conducted in ten commonly used self-driving sequences without any special selection. Compared to directly modeling the transformation without any constraint, our spatiotemporal constrained pose refinement leads to higher rendering quality of the reconstructed NeRF with more accurate camera poses. Fig. 12 of our revised paper also illustrates the improvement of rendering from this constraint.

Table 3. Effectiveness of spatiotemporal constraint tested on ten Waymo scenes.

9) Reference on ZipNerf.

Thanks. We have cited the reimplementation of Zip-NeRF that we use in the revised paper.

[1] Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM, 1981, 24(6): 381-395.

[2] Chen Y, Lee G H. DBARF: Deep Bundle-Adjusting Generalizable Neural Radiance Fields[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 24-34.

[3] Smith C, Du Y, Tewari A, et al. FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow[J]. Advances in Neural Information Processing Systems, 2023

Thank you very much for your valuable feedback. Here is our reply.

• Thank you for your correction. We have revised our response to "before".

• We appreciate your mention of the challenge posed by dynamic objects. Pose estimation under dynamic scenes is a specialized task, where many solutions are based on key points. For example, Mask-SLAM[1] and DynaSLAM[2] filter out key points in areas that may be dynamic objects (vehicles and pedestrians) through semantic segmentation. As our paper mainly addresses the challenges brought by introducing multiple cameras, we did not initially consider the issue of dynamic objects. But the solutions for filter dynamic objects, such as the semantic segmentation, can be easily added to our pose optimization module.

Finally, we truly appreciate your recognition and suggestions for our work.

[1] Kaneko M, Iwami K, Ogawa T, et al. Mask-SLAM: Robust feature-based monocular SLAM by masking using semantic segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2018: 258-266.

[2]Bescos B, Fácil J M, Civera J, et al. DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes[J]. IEEE Robotics and Automation Letters, 2018, 3(4): 4076-4083.

4) The experiment comparisons are limited since only Mip-NeRF was used. Why not compare to some large-scale NeRFs such as block-NeRF or multi-views NeRFs such as MC-NeRF[5] and NeRF-MS[6].

Thanks for your suggestion. We have compared the strategies used for color correction and pose refinement in Block-NeRF in Tab. 5, Tab. 6, Fig. 11, and Fig. 12 of our revised paper (in Appendix A.3.2). Block-NeRF does not release its code. According to its paper, it is built on Mip-NeRF. For a fair comparison, we replace our color correction strategy and pose refinement strategy in ZipNeRF with the strategies used in Block-NeRF. (Block-NeRF uses the color correction strategy from NeRF-in-the-wild and pose refinement strategy same as NeRF$--$.)

We add an experiment in Tab. 2 below, demonstrating our method's superiority over NeRF-MS. NeRF-MS lacks specific modeling of color correction for multi-camera systems, potentially correcting appearance unrelated to color inconsistency and resulting in inferior performance. MC-NeRF is a contemporaneous work posted on arXiv in September 2023 without public codes. Therefore, we cannot compare it with ours within the limited discussion time. It focuses on optimizing the intrinsics of different cameras during pose optimization while we propose the spatiotemporal constraint between different cameras to enhance pose optimization. The discussion of these works is also updated in the related work of our revised paper.

Table 2. Comparison of our method with NeRF-MS in a randomly selected Waymo scene.

[1] Ye S, Lu S P, Munteanu A. Color correction for large-baseline multiview video[J]. Signal Processing: Image Communication, 2017, 53: 40-50.

[2] Lu S P, Ceulemans B, Munteanu A, et al. Spatio-temporally consistent color and structure optimization for multiview video color correction[J]. IEEE Transactions on Multimedia, 2015, 17(5): 577-590.

[3] Mildenhall B, Hedman P, Martin-Brualla R, et al. Nerf in the dark: High dynamic range view synthesis from noisy raw images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 16190-16199.

[4] Wang P, Zhao L, Ma R, et al. BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 4170-4179.

[5] Gao Y, Su L, Liang H, et al. MC-NeRF: Muti-Camera Neural Radiance Fields for Muti-Camera Image Acquisition Systems[J]. arXiv preprint arXiv:2309.07846, 2023.

[6] Li P, Wang S, Yang C, et al. NeRF-MS: Neural Radiance Fields with Multi-Sequence[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 18591-18600.

1) Why we can not use some classical multiple views color correction solutions in the structure-from-motion field as a pre-processing step instead of a module inside the NeRF?

First, limited overlap in images from different cameras hampers classical color correction methods, still causing color inconsistencies in NeRF training. Secondly, traditional multi-view color correction methods [1,2] propagate the reference colors to the corresponding 2D pixels in the target frames, which can be challenging to ensure global color consistency across a large set of images. However, NeRF can directly learn the color correction in the 3D space, thus naturally ensuring global color consistency in the rendered 2D images.

2) What are the main differences between the proposed color correction method and existing works, such as Block-NeRF and Raw-NeRF[3]?

Existing works, including Block-NeRF, alleviate the inconsistent color supervision by modeling image-dependent appearance with a global latent code for each image, which does not perform well in our setting (as illustrated in Fig. 11 and Tab. 5 of our paper). Compared to them, (i) we propose the concept of layer-wise color correction, which approximates color correction in foreground and sky regions as different affine transformations. This not only avoids the disentanglement of attributes unrelated to color inconsistency when the latent code implicitly models the image-dependent appearance, but also ensures the ability to model different color transformations in different regions of a single image. (ii) we provide a novel module ``Virtual Warping'' to further improve color correction. Learning one color correction for each image leads to overfitting when the training images lack color and viewpoint diversity. Virtual warping provides diverse color observations from novel viewpoints of different cameras for each 3D region, alleviating the overfitting of color correction and improving the 3D recovery.

Raw-NeRF inputs the noisy mosaicked linear raw images from the sensor rather than the common images processed by a camera pipeline. Thus, it cannot be directly applied to our scenario.

3) What is the novelty of the proposed method over existing NeRF-pose joint optimization methods, such as BARF, and BAD-NeRF[4]?

Existing methods, including BARF and BAD-NeRF, jointly optimize poses with NeRF by photometric loss. However, they do not consider two difficulties of pose optimization in multi-camera systems.

(i) Images taken by multiple cameras exhibit noticeable color inconsistencies. Joint optimization of NeRF, color correction, and poses based on photometric loss only might not provide sufficient constraints. Therefore, we optimize the poses more directly by establishing pixel correspondences. We add an experiment to demonstrate our superiority to BARF in Tab. 1 below. BAD-NeRF is tailored for the rendering of blurring images, which is not comparable with our method.

Table 1. Comparison of our pose refinement method with BARF. Tested on a randomly selected Waymo scene.

(ii) Inaccurate relative transformations between different cameras will cause rendering artifacts. However, directly optimizing the camera transformations without any constraint is unstable and inaccurate in our setting, since the images captured by multiple cameras share limited overlaps. We introduce a constraint that explicitly models the relative transformations between cameras as temporally consistent but optimizable variables to achieve more accurate pose refinement.

1) The share of our code.

We promise to share our code upon acceptance.

2) The formulation of the loss.

$L_{rpj}$ is used for pose refinement only. The total loss for NeRF training is the weighted sum of the photometric loss, $L_{sky}$ and $L_{reg}$. We add the formulation of the loss and revise the training strategy in Sec. 3.5. Please check out general response 1 about training strategy and loss formulation.

3) The definition of $\mathbf{b}$, $\mathbf{d}$, $d_v$, $d_o$.

Many thanks. In Eq. 5, $\mathbf{b}$ and $\mathbf{d}$ should be $\mathbf{x}$ and $\mathbf{y}$ respectively, which represent the translation term of the color correction matrix. In Eq. 6, $d_v$ and $d_o$ refer to the pixel depth in virtual views and the corresponding pixel depth in original real views. All required definitions have been added to the paper.

Thanks for your reply and understanding. We will definitely share the code.

5) The relationship and difference to the bundle adjustment.

Bundle adjustment (BA) is a widely used technique that involves simultaneously adjusting pose parameters and 3D point locations. In our framework, we perform BA to optimize poses, and additionally incorporate the spatiotemporal constraint between cameras to eliminate errors of the under-constrained pose estimation of small-overlapping cameras in a multi-camera system.

6) How much does the time cost to filter out inaccurate depths through a geometric consistency check?

The geometric consistency check takes about 0.3 seconds per depth map. Thus, for a 300-image scene, it could be done in 90 seconds.

1） The whole pipeline seems verbose since three independent modules are stitched together with few connections. Can the proposed UC-NeRF be trained in an end-to-end manner?

Yes, our NeRF is trained end-to-end with the learnable color correlation and virtual warping as data augmentation. The pose refinement is performed as a preprocessing step to provide accurate poses for NeRF training. Our method presents a unified framework for solving the problems of neural rendering with under-calibrated multi-view cameras, where the three modules are designed to tackle the key problems in this setting, such as inaccurate poses, and inconsistent color images. The proposed three modules can complement each other and contribute as a whole to improve the final rendering results, which are correlated with each other and effective.

2） The efficiency comparisons of different methods.

Thanks for your suggestion. We add Tab. 1 below to the paper for efficiency comparison. Note that our training time includes both steps (pose refinement and NeRF training) described in Sec. 3.5. Zip-NeRF is more efficient than other methods except Instant-NGP, which is specifically designed for NeRF acceleration. Since our method is built upon Zip-NeRF, our method consumes a bit more time than Zip-NeRF but achieves a significant improvement in rendering quality.

Table 1. Efficiency Analysis. Tested on one NVIDIA Tesla V100 GPU with image resolution $1920 \times 1280$.

3） The highlight of the first two contributions.

The first two modules are embedded in the NeRF framework for end-to-end training. They respectively address a specific challenge of color correction during training of NeRF in multi-camera systems, i.e., (i) the way to model color correction and (ii) color correction in limited observations.

(i) Existing methods do not perform well in color correction in our setting, as shown in Fig. 11 and Tab. 5 (in Appendix A.3.2) of our revised paper. They generally alleviate the inconsistent color supervision by modeling image-dependent appearance with a global latent code for each image. However, the lack of explicit modeling for color correction from the latent code can lead to the disentanglement of attributes unrelated to color inconsistency. Moreover, the capacity of a global latent code to uniformly correct colors in different regions of an image is limited, especially when different regions correspond to different color transformations. We propose the concept of layer-wise color correction. We approximate color corrections in foreground and sky regions as different affine transformations. This strategy not only avoids the disentanglement of attributes unrelated to color inconsistency when the latent code implicitly models the image-dependent appearance, but also ensures the ability to model different color transformations in different regions of a single image.

(ii) The limited observations used for color correction. Learning one color correction for each image leads to overfitting when the training images lack color and viewpoint diversity, especially for areas observed by side-view cameras, which have fewer observations and limited overlapping with front-view areas. Virtual warping provides a novel way to improve color correction. It provides diverse color observations from more viewpoints of different cameras for each 3D region, alleviating the overfitting of color correction and improving the 3D recovery.

4) The performance under diverse weather conditions.

Thanks for your suggestion. Our datasets have included sunny and overcast weather, and we add rainy and night conditions here. As shown in Tab. 2 below, our approach notably improves across these scenarios. Please check the rendered images in the Fig. 13 (in Appendix A.3.2) of the revised paper and the video in the supplementary material.

Table 2. Comparison of diverse weather conditions. Tested on Waymo Segment-100170 (Rainy) and Waymo Segment-150908 (Night).

Thanks for your feedback. We greatly appreciate your questions and suggestions concerning our pose optimization module. First, we would like to emphasize that our UC-NeRF is designed to address the problem of color inconsistency in NeRF training under a multi-camera system, and pose refinement is only a part of this. It works in conjunction with other modules to tackle the challenges of color inconsistency.

In our spatiotemporally constrained pose refinement module, we optimize the poses using bundle adjustment (BA). However, BA is a general pose optimization technique that relies on different energy functions to optimize poses. Many works enhance pose accuracy through the design of energy functions, such as Pixel Perfect SFM[1] rewrites reprojection loss as pixel's feature loss, and NIM-SLAM [2] rewrites pixel's photometric loss as multi-scale patch’s photometric loss. For our multi-camera setting, we also redesign the energy function. We consider the prior of the cameras, i.e., their spatiotemporal relationships, and add additional constraints to the poses. We design the energy function in the form of Eq.1 below:

$\mathbf{T}^{i}_{k}$, which is the kth camera's pose at time $i$, is modeled as $\mathbf{T}^i \Delta \mathbf{T}_k$ with two parts: the ego pose $\mathbf{T}^i$, and the temporally consistent and optimizable relative transformation of the cameras $\Delta \mathbf{T}_k$. The inclusion of this constraint significantly improves the rendering results, as shown in Tab. 1 below and Fig.12 in the appendix of the paper.

Table 1. Effectiveness of spatiotemporal constraint tested on ten Waymo scenes.

Otherwise, independently modeling camera poses without the proposed spatiotemporal constraint is feasible, but it may lead to less accurate pose estimation. This is primarily because images captured by a moving vehicle share limited overlaps, especially among the images captured by different cameras. This could potentially result in insufficient connections between images taken by different cameras, making the pose optimization relatively less constrained and unstable.

In addition, we want to state the motivation for using BA. We have tried various strategies for joint optimization of NeRF and pose in multi-camera systems. However, due to the color differences between images, joint optimization of NeRF, pose, and color correction through photometric loss only might not provide sufficient constraints. Tab. 2 below (also shown as Tab. 6 in our paper) also demonstrates that explicitly establishing pixel correspondence to optimize poses shows more significant improvement than jointly learning poses, NeRF, and color correction.

Table 2. Comparison of our pose refinement method with nerf-pose joint refinement methods BARF, NeRF-- (the same pose refinemment strategy in Block-NeRF). Tested on a randomly selected Waymo scene.

[1] Lindenberger P, Sarlin P E, Larsson V, et al. Pixel-perfect structure-from-motion with featuremetric refinement[C]//Proceedings of the IEEE/CVF international conference on computer vision (ICCV). 2021: 5987-5997.

[2] Li H, Gu X, Yuan W, et al. DENSE RGB SLAM WITH NEURAL IMPLICIT MAPS[C]//The Eleventh International Conference on Learning Representations (ICLR). 2022.

1) The title needs to be more specific.

Thanks for your suggestion. Our method aims at novel view rendering with under-calibrated multi-camera systems, where autonomous driving is one of the typical scenarios. Moreover, public autonomous driving datasets like Waymo are widely used benchmarks, making it convincing to compare our method with other works using the datasets. We are also open to incorporating ''autonomous driving'' in the title if needed.

2) The missing definition of ‘b’ and ‘d’.

Thanks for pointing this out. b and d should be x and y as the translation term in the color correction matrix. Please check our general response 2 about the typos and symbols.

3) Isn't it better to reduce reprojection error as a function of direct 4x4 transformation between the cameras across time?

Thanks for your suggestion. As you pointed out, the relative pose between cameras may change during driving, but such changes are typically caused by accumulated shakes over a vehicle's long-term driving like several months. NeRF models generally operate on very short video sequences lasting from minutes to hours, during which the changes among the relative poses of the multiple cameras are subtle. On the contrary, directly modeling the transformation between cameras without the proposed spatiotemporal constraint is feasible, but it may lead to less accurate pose estimation. This is primarily due to the fact that images captured by a moving vehicle share limited overlaps, especially among the images captured by different cameras. This could potentially result in insufficient connections between images taken by different cameras, making the pose optimization relatively less constrained and unstable. Considering both factors, we propose to leverage the spatiotemporal constraint to enhance the pose refinement.

We add an experiment in Tab. 1 below to analyze the spatiotemporal constraint. The experiment is conducted in ten commonly used self-driving sequences without any special selection. Compared to directly modeling the transformation without any constraint, our spatiotemporal constrained pose refinement leads to higher rendering quality of the reconstructed NeRF with more accurate camera poses. Fig. 12 of our revised paper also illustrates the improvement of rendering from this constraint.

Table 1. Effectiveness of spatiotemporal constraint tested on ten Waymo scenes.

4) How does the segmentation model perform in virtual views?

We do not employ any segmentation in virtual views. Both the sky mask and color of a pixel in the virtual view are obtained from the corresponding pixel in the original frames of the input sequence based on the warping by depth.

5) The training strategy is not very clear. Sec. 4.4 is redundant. Explain Sec. 3.5 in detail.

Thanks for your suggestion. We revise the training strategy in Sec. 3.5 (detailed in general response 1) and move Sec. 4.4 to the appendix.

6) The green region adjoining the bright sky appears to be hazy.

Thanks for the observation. The observed blurriness may stem from some random factors, such as sky segmentation errors or insufficient supervision of the regions due to few samplings. We have noticed similar blurriness in the rendering results of Zip-NeRF, whereas our results are clearer, as shown in Fig. 13 of our revised paper. On the whole, our rendering quality considerably outperforms that of Zip-NeRF in numerous areas, like consistent colors, sharp details, and clear logos.

Thanks for your valuable feedback. We greatly appreciate your suggestion regarding the title and agree with it. Accordingly, we have incorporated the concept of 'autonomous driving' into the title in the most recent revision of our paper.