LuSh-NeRF: Lighting up and Sharpening NeRFs for Low-light Scenes

W1 Part1: Error in Eq.4 & 5 and the missing derivation.

R: We thank the reviewer for the careful reading and apologize for the mistakes in Eq.5, which omitted the deblurring process. The missing derivation is also added to Eq. 5. The revised Eq. 5 should be: $C_{noisy}(r) = CTP(C_{S-NeRF}(r)) + C_{N-NeRF}(r) = CTP(\sum_{i=1}^{N} w_i c_i) + n_{\frac{N}{2}}, \quad where \quad n_{\frac{N}{2}} = MLP_{N-NeRF}(P_{mid}, d),$

where the $P_{mid}$ and $d$ is the intermediate point coordinate and the view direction. The $\text{CTP}(\cdot)$ function is the CTP module, which is illustrated in the Subsec. 3.3.

W1 Part2: Inproper name for N-NeRF.

R: We refer to it N-NeRF as it is a multilayer MLP network similar to the S-NeRF, and it is ray-conditioned as the inputs are the view direction $d$ and the coordinates of the mid-point (fixed for each ray) during the whole training process. We will revise its name to "Noise Estimator" in the revision to avoid any ambiguities.

W2: Insufficient novelty of CTP module.

R: Note that DP-NeRF [19] does not consider the frequency domain information. DP-NeRF and other works directly uses the rays of all image regions to predict the camera trajectory, which is severely interfere by the low light noise (see Fig. 6(b) for details).

The novelty of our CTP module lies in that it learns to identify low-frequency-dominated regions that are more robust to noise for kernel prediction. With these regions, LuSh-NeRF can significantly reduces the influence of noise on camera trajectory modeling. Some ablation experiments is in the table below.

From the table, it can be derived that the CTP module that utilizes the help of low frequency information can be better employed in low light scene.

W3: Excluding the edge areas.

R: The reviewer is correct that blurring is more noticeable in high-frequency areas. However, in our task, noise may also significantly affect the images, making the high-frequency information unreliable for blur kernel predictions.

Our CTP module is proposed to minimize the negative influence of these low-quality regions for blur modeling. As shown in Fig.2 in the rebuttal PDF, the main edges in the image can be effectively preserved, while the severely disturbed regions (e.g., the grass and the sky) are discarded in the optimization process.

The tables below and in R3Q2 show the quantitative results of using different frequency filter radius $r$ and intensity threshold $T$, based on which we set the radius to 30. Will discuss more in the revision.

W4: Correct to start training N-NeRF after 60K iters?

R: When $\beta=0$, N-NeRF in SND module is still co-optimized with S-NeRF for noise estimation. When $\beta$ is turned on after 60K iterations, $L_{consistency}$ will reinforce the consistency across different views, which further facilitates the noise estimation of N-NeRF.

If $L_{consistency}$ is used at the beginning of the training phase, the Image Matching method cannot accurately align the images due to the low quality of the images rendered by the S-NeRF in the early training stage, which may deteriorate the performance and extend the training time.

We report the results of turning on $\beta$ at different iterations as references:

As shown in the table, adding $L_{consistency}$ at 60k training iteration yields better results.

W5: The generalizability of the image-matching method.

R: We empirically find the pre-trained GIM[40] has great generalization to cross-domain data. Some of the matches obtained by GIM on S-NeRF rendered results are shown in Fig.3 in the rebuttal PDF to demonstrate its generalizability.

W6: Insufficient Dataset Contribution.

R: Actually, we did not just take images from the LOL-Blur dataset and then ran an off-the-shelf SFM method to obtain the camera parameters. To build an effective dataset, we did the following works:

(1) Scene selection: we go through the whole LOL-Blur dataset to select scenes featuring different environments (covering indoor/outdoor situations) with different camera motions and lighting.

(2) Image selection: we manually select 20-25 (out of around 60) images per scene with relatively high-quality for the estimation of camera poses, and ensure their luminance is close to imitate the real shooting situation.

(3) Camera Pose Estimation: Note that the estimation of camera pose by COLMAP for low-light blurry images are often unreliable. We first repeat the estimation of COLMAP 30 times for each scene to select the optimal pose result, and then manually tune it with our baseline NeRF model to improve the accuracy.

Our experimental results show that the resulting dataset helps us learn the robust LuSh-NeRF for handling low-light blurry scenes. We will clarify these in the revision.

W7&8: Missing references and typos.

R: As suggested, we will cite and discuss these papers, and correct the typos in the revision.

W1: Insufficient Dataset Contribution.

R: Actually, we did not just take images from the LOL-Blur dataset and then ran an off-the-shelf SFM method to obtain the camera parameters. To build an effective dataset, we did the following works:

(1) Scene selection: We went through the whole LOL-Blur dataset to select scenes featuring different environments (covering indoor/outdoor situations) with different camera motions and lighting.

(2) Image selection: We manually select 20-25 (out of around 60) images per scene with relatively high-quality for the estimation of camera poses, and ensure their luminance is close to imitate the real shooting situation.

(3) Camera Pose Estimation: Note that the estimation of camera pose by COLMAP for low-light blurry images are often unreliable. We first repeat the estimation of COLMAP 30 times for each scene to select the optimal pose result, and then manually tune it with our baseline NeRF model to improve the accuracy.

Our experimental results show that the resulting dataset helps us learn the robust LuSh-NeRF for handling low-light blurry scenes. We will clarify these in the revision.

W2: Unclear notation.

R: $n_{\frac{N}{2}}$ represents the noise value obtained by the N-NeRF MLP structure, and the $\frac{N}{2}$ means the middle sampled points on each ray.

We thank this reviewer for pointing out this issue, and will revise the Method section to improve the clarity in the revision.

W3: Relation of Equations 4 and 5.

R: We apologize for the confusion caused by Eq.5, as it omits the deblur process in LuSh-NeRF, and the revised Eq.5 should be: $C_{noisy}(r) = CTP(C_{S-NeRF}(r)) + C_{N-NeRF}(r) = CTP(\sum_{i=1}^{N} w_i c_i) + n_{\frac{N}{2}}, \quad where \quad n_{\frac{N}{2}} = MLP_{N-NeRF}(P_{mid}, d),$

where the $P_{mid}$ and $d$ is the intermediate sampling points coordinate and the view direction. The $\text{CTP}(\cdot)$ function is the CTP module, which is illustrated in the Subsec. 3.3.

W4: Relation between CTP module and DP-NeRF.

R: The CTP module adopts the Rigid Blurring Kernel (RBK) module of DP-NeRF. The RBK module uses two MLPs to model the Rigid Camera Motion parameters for each viewpoint in the dense SE(3) field, one MLP to calculate the weights of the rendered pixels obtained from each camera pose within the kernel, then these pixels are used to compute the final blurry pixel values via weighted sum operation.

The difference between the CTP and the RBK module lies in that, the frequency-domain information is introduced into the kernel estimation to help achieve more accurate blur modeling by exploiting the image regions which are less affected by the low light noise. We will clarity them in the revision.

Q1: Why is the noise modeled with a NeRF? Why not optimize ...

R: Thanks for the question. We refer to it N-NeRF as it is a multilayer MLP network similar to the S-NeRF. We will revise its name to "Noise Estimator" in the revision to avoid any ambiguities.

The N-NeRF is ray-conditioned as the inputs are the view direction $d$ and the coordinates of the mid-point (fixed for each ray) during the whole training process.

As suggested, we try the view-dependent inputs $(x,y,N)$, i.e., a 2D coordinate $(x,y)$ with the view id $N$ as the inputs to the N-NeRF to optimize a noise tensor per view. The results are reported below and the performance difference is quite small, which shows that both strategies work in our task.

Q2 Part1: The importance of the low-pass filter.

R: We have shown in the Fig. 2 in rebuttal PDF the difference between the two masks obtained by directly performing a rgb intensity threshold and performing a Frequency filtering before taking a threshold.

(1) The RGB intensity of many noise-dominant regions are also high (e.g., the sky and the grass), which are harmful to the blur estimation but not excluded in the mask produced by thresholding.

(2) The CTP module uses a Frequency filter to identify the low-frequency-dominated regions of the image, and then obtains the desired mask based on the RGB intensity values. The gradients of rays in high-frequency and/or dark regions (regions more severely affected by noise) are detached during the blur kernel optimization process, which ensures better blur modeling.

Q2 Part2: Just threshold the image without the low-pass filter.

R: We have conducted the ablation experiments of CTP module on two synthetic scenes in the following table.

The experimental results show that the frequency filter provides a more desirable mask, which is superior to the directly obtaining mask with the RBG intensity thresholds.

Thank you for the response to my comments.

W1: Insufficient Dataset Contribution.

For the discussion on the dataset, I am still not satisfied, I would not attribute the paper any significant contribution for selecting views for 5 synthetic and 5 captured scenes from an existing dataset and estimating the camera parameters, even though I appreciate COLMAP is not always straightforward to use.

W4: Relation between CTP module and DP-NeRF.

Thank you very much for this explanation, that helps me better understand the difference between the two modules. But frankly, I think even this explanation is not enough for what the RBK module exactly does, I think its common practice to explain properly the methods used if they are non-standard even if they come from other papers. I would implore the authors to add a more thorough description, even if just in the supplement.

Q1. Why is the noise modeled with a NeRF?

This begs the question, why the authors don't use the (x, y, N) based formulation. Isn't this formulation more efficient and understandable? Presumably, this MLP is smaller, and since there is no spatial consistency required, it makes more sense to use it than a 3D-conditioned MLP?

EDIT: Maybe I am misunderstanding something, what is the ray midpoint the author's query? This changes as per the random sampling, is that right? So for the same pixel, I could be querying a different midpoint in different iterations?

Otherwise, I thank the authors for their responses, I am satisfied with the answers provided to all other questions. I think the discussion on the low-pass filter is especially useful, since the Freq. Domain Thresholding is one of the main contributions to the CTP module. I would implore the authors to add these results to the paper.

Thanks for your reply and we are glad to see that our response can address most of your raised concerns. We would like to futher clarify the below issues.

R3W1: Insufficient Dataset Contribution.

R: (1) The dataset contribution is one part of our 3rd contribution and we still have other technical contributions.

(2) Note that our dataset is necessary as we are handling a new task (handling NeRF in low-light scenes with camera motions). To construct this dataset, it took us more than 3 full weeks to select images and tune the camera pose parameters. Note that the colmap-free method [A] does not handle our task well as it is very difficult to optimize the camera pose directly in low-light blurry scenes.

(3) We can revise our third contribution to emphasize more on the experimental evaluations and state-of-the-art results of our model. However, we do need this dataset for evaluation and we did put efforts into constructing this dataset.

[A] NeRF--: Neural radiance fields without known camera parameters, arXiv:2102.07064, 2021.

R3W4: Relation between CTP module and DP-NeRF.

R: Thansk for your suggestion. The Rigid Blurring Kernel (RBK) module in DP-NeRF[19] models the scene blur kernel by simulating the 3D camera motions via the following two main parts:

Ray Rigid Transformation (RRT): The RRT models the blurring process of an image. It is formulated as ray transformation derived from the deformation of rigid camera motion, which is defined as the dense SE(3) field for scene $s$ and modeled by the MLPs as:

$S_s = (r_s;v_s) = (\mathcal{R}(\mathcal{E}(l_s);\mathcal{L}(\mathcal{E}(l_s))), where \ s \in view_{img},$

where $l_s$ is the latent code for each view through the embedding layer in [B], $\mathcal{R}, \mathcal{L}, \mathcal{E}$ are three MLP networks, $view_{img}$ is the training view set. The $S_s = (r_s;v_s) \in \mathbb{R}^6$ is the matrix which will be used for the RRT modeling as follows:

$ray_{s;q}^{RRT} = Rigid-Transform(ray_s, (r_s;v_s)),$

where $Rigid-Transform()$ function is the standard 3D rigid transformation operation, $ray_s$ and $ray_{s;q}^{RRT}$ are the orgin ray and the transformed rays in scene $s$, $q \in \\{1,...,k \\}$, $k$ is a hyper-parameter that controls the number of camera motions contributing to the blur in each scene $s$. The blurry RGB value at $ray_s$ can be acquired by weighting sums of the NeRF volume rendering values $C_{s;0}$ and $C_{s;q}$ from $ray_s$ and $ray_{s;q}^{RRT}$.

Coarse Composition Weights: The Coarse Composition Weights are computed for each ray obtained by the RRT:

$m_{s;0,1,...,k} = \sigma(\mathcal{W}(\mathcal{E}(l_s))), where \ \sum_{i=0}^{k}m_{s;i} = 1,$

where $m_s$ is the final weight for each ray in RRT. Finally, blurry color $\mathnormal{C}_s$ for scene $s$ can be computed by the weighted sum operation as shown below:

$C_s = m_{s;0}C_{s;0} + \sum_{q=1}^{k}m_{s;q}C_{s;q}.$

We will add these information to the revision.

[B] Optimizing the latent space of generative networks, arXiv:1707.05776, 2017.

R3Q1: Why is the noise modeled with a NeRF? Why not optimize ...

R: We agree with the reviewer that the $(x,y,N)$-based formulation is another possbile implementation of modeling noise comparing to our current one (the rendering results of both do not differ much). Note that this does not affect our goal of decomposing the scene and noise information.

Regarding the ray midpoint, we uniformly sample (instead of random sampling in S-NeRF) between the near and far fields (calculated by COLMAP) of the ray, and select the coordinates $P_{mid}$ of the intermediate sampling point, along with the view direction $d$, as the input of N-NeRF. Since the bounds of the ray do not change during the training phase, the midpoints for each ray (one-to-one map with pixels) is same.

We will incorporate all these information into our revision.

W1: Ablation studies regarding the roles of proposed modules.

R: Thanks for your positive feedback on our work, the visualization of the ablation experiments can be found in Fig.6 in the main text. To better demonstrate the effectiveness of the different modules in LuSh-NeRF, we performed quantitative ablation experiments on all the synthetic datasets in the table below:

The specific analysis is as follows:

(1) From Line 1 and 2, the ScaleUp preprocessing enhances the NeRF's reconstruction capabilities, resulting in improved rendering results.

(2) From Line 3 and 4, CTP leverages frequency domain information to refine Blur Kernel predictions, boosting the perceptual quality of reconstructed images. However, this process diminishes PSNR and SSIM results due to neglecting noise interference.

(3) From Line 2 and 5, the SND module can disentangle noise and scene information from input noisy-blurry images, leading to substantial improvements in PSNR and SSIM metrics. However, the SND module is not capable of resolving the blur problem, which leads to an minor improvement in the image's perceptual quality (more important for the rendered images).

(4) From Line 4, 5, 8, the combined application of SND and CTP modules enhances image perceptual quality and outperforms the sole use of CTP in terms of PSNR and SSIM and SND in terms of LPIPS.

(5) From Line 7, 8, it can be concluded that decoupling the noise first, and then modeling the scene blur is a more robust restoration order, which can effectively reduce the interference of noise in the deblurring process, and obtain better performance metrics.

W2: Colmap-free NeRF methods may be more effective?

R: The authors highly agree with your ideas. Colmap-free NeRF methods robust to low-light and blur phenomena would be much more helpful for the problem we propose. However, existing colmap-free NeRF methods may not handle our task easily.

The table below compares to a COLMAP-Free NeRF [1]. The results demonstrate the existing COLMAP-free NeRF method cannot effectively handle low-light scenes with motion blur, due to the inaccurate poses optimized from the image directly. Exploring colmap-free methods for our task can be interesting for future work.

[1] Wang Z, Wu S, Xie W, et al. NeRF--: Neural radiance fields without known camera parameters[J]. arXiv preprint arXiv:2102.07064, 2021.

Q1: Reduce the memory size of the paper.

R: Thanks for the suggestion, we will compress the images and reduce the PDF size in our revision.

Q2: More comparison with Aleth-NeRF.

R: As suggested, we have performed several experiments with Aleth-NeRF [7] on our proposed synthetic and realistic scenes. The visualization results of one comparison in this experiment are shown in the Fig.1 in Rebuttal PDF.

As Aleth-NeRF focuses on adjusting the luminance of the scene, it cannot handle the blur in low-light scenes. Will include more in the revision.

Thank you for your response. My concerns have been fully resolved, and this is an excellent work. I improve my rank to weak accept.

W1: Ablation studies with quantitative results.

R: As suggested, we have conducted detailed ablation studies on all the synthetic scenes in the following table:

(1) From Line 1 and 2, the ScaleUp preprocessing enhances the NeRF's reconstruction capabilities, resulting in improved image.

(2) From Line 3 and 4, CTP leverages frequency domain information to refine Blur Kernel predictions, boosting the perceptual quality of reconstructed images. However, this process diminishes PSNR and SSIM results due to neglecting noise interference.

(3) From Line 2 and 5, the SND module can disentangle noise and scene information from input noisy-blurry images, leading to substantial improvements in PSNR and SSIM metrics. However, the SND module is not capable of resolving the blur problem, which leads to an minor improvement in the image's perceptual quality (more important for the rendered images).

(4) From Line 4, 5, 8, the combined application of SND and CTP modules enhances image perceptual quality and outperforms the sole use of CTP in terms of PSNR and SSIM and SND in terms of LPIPS.

(5) From Line 7, 8, it can be concluded that decoupling the noise first, and then modeling the scene blur is a more robust restoration order, which can effectively reduce the interference of noise in the deblurring process, and obtain better performance metrics.

W2: Mistakes in Tab.1.

R: Thanks for your careful reading, we will correct these mistakes in the revision.

W3: Missing Citations.

R: Thanks for your valuable reviews. HeadNeRF [1] integrates NeRF to the parametric representation of the human head. MoFaNeRF [2] propose the first parametric model that maps free-view facial images into a vector space with NeRF. INFAMOUS-NeRF [3] propose a novel photometric surface constraint that improves the face rendering performance.

These inspiring works are of importance to the Human NeRF development. As suggested, we will cite and discuss these works in our revision.

[1] Hong, Yang, et al. "Headnerf: A real-time nerf-based parametric head model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Zhuang, Yiyu, et al. "Mofanerf: Morphable facial neural radiance field." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

[3] Hou, Andrew, et al. "INFAMOUS-NeRF: ImproviNg FAce MOdeling Using Semantically-Aligned Hypernetworks with Neural Radiance Fields." arXiv preprint arXiv:2312.16197 (2023).