Rad-NeRF: Ray-decoupled Training of Neural Radiance Field

R4-Q1: My main concern is the novelty. Although this work aims for different tasks, it is quite like Switch-NeRF.

Thanks for this question. In our opinion, the most fundamental difference between Switch-NeRF and our Rad-NeRF is that Rad-NeRF is a ray-based multi-NeRF framework (the first ray-based framework as far as we know), while Switch-NeRF is a point-based multi-NeRF framework. And these two types of designs target different objectives. Specifically, Switch-NeRF decomposes 3D points to different sub-NeRFs to model larger scenes, and Rad-NeRF decomposes rays to tackle the training interference from invisible rays, which is especially of concern in complex scenes.

Figure 5 and our newly added Figure R.1 show that scaling the number of sub-NeRFs by decoupling rays to different sub-NeRFs offers better performance-parameter scalability than scaling by decomposing spatial positions or scaling the MLP or spatial grid. We believe that in future applications of scaling NN-based 3D modeling methods for complex scenes, our newly proposed dimension (scaling the number of sub-NeRFs by decoupling rays) is a worth-considering dimension in the compound scaling strategy. Besides the major novelty in designing the ray-based multi-NeRF framework, we would also like to mention our introduction of mutual depth supervision between sub-NeRFs. This technique is simple but effective and can be easily adopted in other multi-NeRF frameworks too, providing a new way of unsupervised geometric regularization without the need for manually defined rules.

R4-Q2: The improved 3DGS nowadays is quite good and does not have the ray sampling problem, so I think this work may have limited influence and help a little for our community.

Thanks for this very worth-discussing topic! We would like to share our thoughts on this topic and welcome any further discussions. Indeed, 3D GS has surpassed NeRF's rendering quality and speed on quite some tasks. Nevertheless, we still think the NeRF framework featured by the parametric neural network representation together with the volume rendering method still has its merits. Maybe one of its most notable strengths lies in its adaptability for generalizable modeling, i.e., in generalizable modeling tasks where we need to leverage the knowledge of existing scenarios to better and more efficiently model some unseen scenarios. In this case, it's very easy to extend the parametric neural network to take in other forms of input other than the position and ray direction. For example, the neural network can take an image, a depth image or even other modalities of information as input to model the current scenario. We think this flexibility is beneficial for pushing towards general 3D modeling or generation.

Besides, there are some scenarios that 3DGS methods cannot handle well, due to the inaccurate and sparsity of point clouds (such as less-textured regions) [1], and the limitation to reason about view-dependent effects (such as reflection and refraction effects)[2]. In these scenarios, NeRF-based methods can offer better visual quality.

[1] PointNeRF++: A multi-scale, point-based Neural Radiance Field

[2] HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces

R3-Q4.2: The analysis of the gating function (Figure 7) lacks depth, and more details would be appreciated.

Thanks for the question! We're not sure does the reviewer means to "add the depth information into Figure 7"? If so, considering that we can directly judge the depth information of the scene by observing the rendered RGB image and thus analyze its correlation with the gating score, we did not add the depth visualization results in Figure 7. As shown in the visualization of Global Response, the gating scores are not always directly related to the scene depth, although some score visualizations show consistency with pixel depths in some views.

If the reviewer is asking for more analyses: We further analyze the additional gating score visualization. As shown in Figure R.3, the gating module can assign different preferences to different sides of the central object. Besides, when the foreground and background of the scene can be clearly distinguished, the preferences of the sub-NeRFs in the fore/background area are also clear (the visualization of view-1 in Figure R.3). However, it is difficult to further summarize the distribution law of the gating scores. Unlike point-based partitioning, the ray-based gating module encodes both the ray origins and directions, incurring higher partition complexity.

R3-Q5: Consideration of performance improvement

Thanks for this question. We'd like to share our thoughts and welcome any further discussions.

Firstly, we regard RadNeRF as an easily integratable and workable strategy to try out in future NeRF applications, as it doesn't require hyperparameter tuning to work well across different types of scenarios, all achieving some improvements (1.02 on mask-TAT, 0.98 on TAT, 0.57 on 360v2, 0.49 on free dataset and 0.82 on ScanNet dataset). For example, in complex scenarios such as the 360v2 dataset, our method can be plugged in and improve the state-of-the-art. Specifically, ZipNeRF obtains 0.74 PSNR gain compared to MipNeRF360. By integrating our method into the ZipNeRF framework, we easily achieve an additional 0.3 PSNR improvement.

Additionally, we compare Rad-NeRF, the first ray-based multi-NeRF framework to our knowledge, with other multi-NeRF training frameworks. The results demonstrate that the ray-based design has distinct advantages over existing point-based designs.

Finally, Figure 5 and our newly added Figure R.1 show that scaling the number of sub-NeRFs by decoupling rays to different sub-NeRFs offers better performance-parameter scalability than scaling by decomposing spatial positions or scaling the MLP or spatial grid. We believe that in future applications of scaling NN-based 3D modeling methods for complex scenes, our newly proposed dimension (scaling the number of sub-NeRFs by decoupling rays) is a worth-considering dimension in the compound scaling strategy.

R3-Q6: The effect of Switch-NGP and Block-NGP

Switch-NGP partitions the scenes in the point dimension, which is fundamentally different from our ray-based multi-NeRF framework. In complex scenes with many occlusions and arbitrary shooting trajectories, Switch-NeRF does not consider the different visibility of a target region to different views and can not tackle the training interference effectively, as noted in the reply to Q1. In addition, Switch-NGP does not handle consistency between multiple sub-NeRFs well due to the lack of overlapping regions and proper regularization between sub-NeRFs.

Block-NGP directly allocates the training images to multiple sub-NeRFs according to the image shooting positions. Judging the relationship between images based solely on shooting positions is insufficient for complex scenes, so it will lead to inaccurate partitioning results and unsatisfactory rendering quality.

We thank the reviewer for the constructive feedback and thoughtful comments. We address the detailed questions and comments below.

R3-Q1: Clarification of Figure.1(c) and motivation

As the reviewer said, NeRF often compensates for RGB appearance with poor geometry. Figure 1(c) might have the correct RGB with inaccurate geometry. This can be seen in the inaccurate sampling probability presented in Figure 1(c), i.e. the PDF distribution of wi in Equation (1). With Figure 1(c), we aim to illustrate that as the NeRF model is apt to learn a poor geometry, undesirable inter-ray training interference can occur during training. Specifically, when rendering ray-3, it is possible to sample the 3D points near the distant object due to inaccurate sampling. Therefore, the modeling of the distant object will be affected by ray-3 color supervision, resulting in the issue of training interference, which is the main motivation of our work.

We're not very sure how the reviewer thinks Figure 1(c) should be changed. Could you please provide more details on this comment? We are eager to understand your perspective better and address any concerns you might have.

R3-Q2: Figure 7 does not support the occlusion hypothesis. I suspect the issue might be related to aliasing, where the pixel footprint differs between the center and boundary regions. This can also explain why MipNeRF360 outperforms the proposed method, and the comparison between ZipNeRF and RadZipNeRF in the supplementary material is very close.

Thank the reviewer for the insightful analysis and careful observation. In the Truck scene, the gating score visualization indeed shows a significant difference between the edge and the central region, correlating with the aliasing issue. We agree that tackling the aliasing issue in some scenarios is an insightful explanation of the Rad-NeRF's effectiveness, which is supplementary to our original motivation targeting scenarios with heavy occlusions. We'll add this discussion to the revision.

That being said, we find that our original motivation for tackling heavy occlusion by decoupling sub-NeRF training is valid. We further provide more gating score visualization results. As shown in Figure R.3 (Global Response), the gating module assigns different preferences to foreground/background regions or the different sides of the caterpillar. Additionally, Figure R.4 shows that when rendering a less-occluded open scene, the gating score exhibits different characteristics and smooth transitions according to the ray directions.

R3-Q3: Will the proposed depth regularization encourage all MLPs to converge to the same output?

The proposed depth-based mutual learning scheme does not encourage all sub-NeRFs to converge to the same output. We provide visualizations of different sub-NeRFs' rendering results in Figure R.5 (Global Response), which validates our analysis and conclusion.

On the one hand, the soft gating module allocates different rays to different sub-NeRFs, making them learn from different views. On the other hand, the depth-based mutual learning scheme only lets sub-NeRFs learn the depth from each other rather than the overall rendered density or RGB distribution.

As for the training curves in Figure.6, Rad-NeRF with more sub-NeRFs converges faster while achieving better rendering quality with the same training iterations. However, as mentioned in the reply to the reviewer tZHj, the rendering performance of Rad-NeRF will gradually saturate as the number of sub-NeRFs increases. Therefore, rad-4 and rad-3 are relatively similar.

R3-Q4.1: Regarding the soft gating, the current input is the ray origin and direction. It might be more effective to include the 3D position, as splitting MLPs based on different spatial regions could be more reasonable.

Thanks for this worth-discussing question. Splitting MLPs based on spatial regions (the point-based multi-NeRF framework discussed in our paper) and splitting MLPs based on rays (as far as we know, RadNeRF is the first attempt) are two orthogonal methods targeting different objectives.

Specifically, our ray-based multi-NeRF framework tackles the training interference from invisible rays, which is especially of concern in complex scenes. Whereas point-based multi-NeRF frameworks that split MLPs based on 3D spatial regions mainly aim to model large scenes by decomposition, which cannot effectively address the training interference issue between invisible rays. For instance, with two sub-NeRFs assigned to 3D points around two objects as shown in Figure 1, ray-3 information pertaining to the central object is still used to train the sub-NeRF associated with the distant object due to inaccurate sampling, potentially resulting in training interference. In contrast, constructing a ray-based gating module-based multi-NeRF framework is a more effective scheme for the training interference issue.

We also conducted a comparison with state-of-the-art point-based multi-NeRF frameworks in Table 1 and conducted the point-based v.s. ray-based ablation study in Table 3. The results show that our ray-based design leads to consistent improvements.

Due to the character limitation, we'll reply to the remaining questions in the comments.

We thank the reviewer for the constructive feedback and thoughtful comments. We address the detailed questions and comments below.

R2-Q1: Clarification of the network structure

Within Rad-NeRF, we adopt a multi-resolution learnable feature grid shared among all sub-NeRFs. Given a 3D point coordinate, we find the surrounding voxels at different resolution levels, index the hash table according to each vertex position, and obtain the final point feature through linear interpolation. During training, loss gradients are backpropagated through the multiple independent MLP decoders, the gating module, and then accumulated in the looked-up learnable feature vectors. We will demonstrate this detail clearly in the revised version.

R2-Q2: Comparison of training and inference time

Following the reviewer's advice, we expand the scalability study in the main paper by supplementing additional results about training time and inference speed. The comparison results are shown in Figure R.1 (Global Response). Compared to the Instant-NGP baseline, all methods for scaling up NeRF's capacity require longer training time and exhibit lower inference speed, including scaling the MLP width and different multi-NeRF frameworks. Among these methods, Rad-NeRF achieves the best tradeoff between training/inference efficiency and rendering quality. Since we adopt a shared feature grid and multiple independent MLP decoders in the Rad-NeRF framework, a point feature needs to be processed by MLPs in turn, which is the major cause of reduced efficiency. However, as multiple independent MLP decoders can be combined into a single MLP through appropriate parameter initialization and freezing, Rad-NeRF can obtain further efficiency improvements and approach the efficiency of scaling the MLP width.

R2-Q3: Does the sub-NeRF cease to update or contribute to the computation if it receives very low soft gating scores?

In the Rad-NeRF training framework, even if a sub-NeRF is assigned an extremely low gating score for a particular ray, it still contributes to the rendering of the scene and remains effectively updated.

Firstly, the gating score represents each sub-NeRF's preference for different rays. Although a sub-NeRF is assigned a low score on a certain ray, it still has preferences and contributes to other rays.

Secondly, we implement CV balancing regularization in the training of Rad-NeRF. This regularization prevents the gating module from collapsing onto a specific sub-NeRF while maintaining the different specialties of each sub-NeRF. Consequently, it is rare for a sub-NeRF to contribute extremely little to the total scene during training.

Thirdly, depth-based mutual learning is proposed as a geometry regularization to keep sub-NeRFs learning from each other. Even if a sub-NeRF is assigned a low gating score and learns little from the ground truth color value, it still learns from the fused depth value and remains effectively updated.

R2-Q4: Are there any automated methods or heuristics for selecting the optimal number of sub-NeRFs based on scene complexity?

We appreciate the reviewer for the insightful and forward-looking question. In this work, considering the limited computing resources, we mainly adopt the default configuration of two sub-NeRFs and extend it to four sub-NeRFs in the scalability study. Intuitively, more complex and larger scenes require more sub-NeRFs, and the rendering performance of Rad-NeRF will gradually saturate as the number of sub-NeRFs increases. We also conduct additional scalability experiments on MaskTAT, a less complex dataset. As the results show, the rendering quality saturates when using three sub-NeRFs, which is similar to the phenomenon observed on the ScanNet dataset, where performance saturation occurs at four sub-NeRFs.

Under such circumstances, determining the optimal number of sub-NeRFs in the Rad-NeRF training framework is an important problem, which we also point out in the conclusion and outlook of the main paper. A possible approach is to adjust the number of sub-NeRFs dynamically during the training process. By setting an appropriate score threshold, sub-NeRFs with average scores lower than the threshold will be removed. Besides, if most sub-NeRFs do not show a significant preference, additional sub-NeRFs can be added.

R2-Q5: The visualization of the gating score of different sub-NeRFs for the same viewpoint or in scenarios without occlusions.

In the visualization results, we adopt two sub-NeRFs in all scenes of the TAT dataset. With this setting, the two sub-NeRFs exhibit complementary gating scores for the same view and we omitted the visualization of sub-NeRF2 for brevity in the main paper. We supplement the visualization results of the other sub-NeRF in Figure R.4 (Global Response).

We further provide the gating score visualization in a less-occluded outdoor scene (Playground scene of TAT dataset). As shown in Figure R.4, when rendering in such an open scene with fewer occlusions, the gating score exhibits different characteristic and smooth transition according to the ray directions. This visualization further validates our analysis that as a 4-layer MLP without sinusoidal position encoding, the gating module incorporates smoothness prior implicitly. For unseen viewpoints, especially in less-occluded outdoor scenes, the gating module exhibits smooth and close scores to the nearest seen view.

We thank the reviewer for the constructive feedback and thoughtful comments. We address the detailed questions and comments below.

R1-Q1: Performance of Rad-NeRF

Overall, compared to the Instant-NGP baseline, our scheme shows improvements in the PSNR metric across various datasets: 1.02 on mask-TAT, 0.98 on TAT, 0.57 on 360v2, 0.49 on free dataset and 0.82 on ScanNet dataset. All the improvements are achieved with the default configuration of only two sub-NeRFs (0.02MB extra parameters). Notably, as the number of sub-NeRFs increases, an additional 0.4 PSNR gain is obtained on the ScanNet dataset. We also check the performance of recently published and related work, such as F2-NeRF, which implements a multi-NeRF framework on Instant-NGP. F2-NeRF claims the PSNR metric improvements of 0.15 on 360v2 and 1.91 on the free dataset, which is on par with the Rad-NeRF effect.

Additionally, on the 360v2 dataset, ZipNeRF obtains 0.74 PSNR gain compared to MipNeRF360. Furthermore, we implement the Rad-ZipNeRF framework (a combined version of Rad-NeRF and ZipNeRF) and achieve an additional 0.3 PSNR improvement. The experimental results prove Rad-NeRF's effectiveness and potential for integration with different single-NeRF frameworks.

Finally, Rad-NeRF goes beyond comparison with existing SOTA single-NeRF methods and extends to the comparison with other multi-NeRF training frameworks based on NGP. In these comprehensive evaluations, Rad-NeRF shows superiority over other multi-NeRF training methods.

R1-Q2: Limitation under the few-shot setting

Previously, we have included the discussion of the limitation under the few-shot setting. This is because that rendering under the few-shot setting presents a greater challenge to both NeRF's and gating module's generalization ability.

We validate Rad-NeRF's performance in the few-shot setting on the LLFF dataset. For 6/9 training views, Rad-NeRF does not exhibit significant benefits or performance improvements compared to Instant-NGP, with all metrics at the same level. This is because insufficient training data affects the training and generalization of the gating module.

However, when rendering with extremely few training data (3 views), Rad-NeRF achieves significantly better rendering quality. We analyze that when training with very few views, the gating module has minimal impact on NeRF's training. Nonetheless, depth-based mutual learning between multiple sub-NeRFs could still exhibit an effective geometric regularization effect, thereby improving rendering performance. This analysis is also validated by the visualization results in the Global Response. As shown in Figure R.2, compared to the baseline, Rad-NeRF reduces the depth rendering ambiguity and shows better geometry modeling in a 3-view setting.

R1-Q3: What is the advantage of Rad-NeRF over Bayes Rays?

As a post-hoc uncertainty assessment framework, Bayes Rays does not change NeRF's training process, only removing "floater" regions corresponding to high uncertainty. However, this solution is not stable and is generally used as an auxiliary solution to improve NeRF's rendering quality.

For example, RobustNeRF treated pixels with larger losses as those with high uncertainty, avoiding the misleading effect of outlier points by discarding the training of those pixels. However, we validated its effect and found that it was difficult to distinguish outlier points from the high-frequency areas that should be learned. Moreover, Instant-NGP regards the spatial points with too low density as regions with high uncertainty and filters these regions when rendering. Although this method works well, it still cannot completely eliminate floaters in difficult scenes and may remove correct regions.

Different from uncertainty-based methods, the proposed Rad-NeRF improves rendering quality by tackling the training interference issue. The depth-based mutual learning method also acts as a geometric regularization to reduce rendering defects. Importantly, Rad-NeRF is essentially orthogonal to these post-training uncertainty removal-based methods and can be integrated with Bayes Rays to obtain further performance improvement. We will incorporate the discussion into the revision.

R1-Q4: Can the method proposed in CR-NeRF solve the problems solved by Rad-NeRF?

CR-NeRF works similarly to style transfer/matching NeRF methods, which are primarily proposed to address the challenges of dynamic appearance caused by different capture times and camera settings. Its goal is to control the style/hue of the rendered image from unconstrained image collections. Differently, Rad-NeRF is proposed to tackle the training interference issue in complex scene rendering and improve the rendering performance in normally shot complex scenes.

CR-NeRF proposes to leverage rays' information for modeling appearance, which is an extension and improvement of the appearance modeling scheme in classical NeRF-W. We used to validate the performance of NeRF-W's appearance modeling method on the issue of floaters before and found nearly no improvement. This is because the problem of floaters and other rendering defects in complex scenes is mainly caused by training interference and inaccurate geometry modeling rather than appearance inconsistency. Compared to CR-NeRF, the proposed Rad-NeRF is more suitable to tackle such challenges.