LeCO-NeRF: Learning Compact Occupancy for Large-scale Neural Radiance Fields

We thank the reviewer for the valuable and insightful comments. We address the detailed questions and comments below.

Q1: Differences between the representations of the binary occupancy and NeRF. Many thanks for this question. We would like to clarify the fundamental differences between the representations of the binary occupancy and Neural Radiance Fields. The representations of MLP, hashgrid and tensoRF can only be used to represent the Neural Radiance Fields in previous methods. They are not used for the representation of the binary occupancy because their occupancy cannot be trained and thus cannot be encoded into the MLP, hashgrid and plane encodings. This is the reason that the KiloNeRF with MLP, the Instant-NGP with hashgrid, the TensoRF with plane features all use a dense grid for the binary occupancy. As far as we know, our method is the first to represent the binary occupancy by MLP and one of our contributions is to make the binary occupancy trainable by the MLP network. Therefore, for "I understand that using a large resolution, dense grid will occupy more memory. In this case, this method should compare with the more eîcient hashgrid representation,", it is not practical to represent the occupancy grid by a hash grid without our training strategies.

For the hash representation of Neural Radiance Fields, we have added new experiments in Table 6 in Appendix C and the table in Reviewer KM5v Q4. In these experiments, our method outperform the Instant-NGP by large margins, i.e. 1.07 point of PSNR on Campus, 0.85 on Building, 0.72 on Rubble and over 0.5 on other scenes. These experiments clearly show that our compact occupancy network can learn much better occupancy than the occupancy grid on different baselines.

We also add an ablation study on the resolution of the dense grid Table 7 in Appendix E and the table below. In this ablation study, we use an occupancy grid of $256^3$ to guide the training of Switch-NeRF. As shown in the table, the occupancy grid with a resolution of 256 (Grid-256) consumes dramatically more time and memory for training. It takes over 46 hours for training of 500K steps, which is much slower than Grid-128, and still gets worse results than ours trained by 14.1h. Moreover, Grid-256 consumes about $5\times$ memory than our method. This study shows clearly the advantage of the compactness of our occupancy network.

Q2: Running time of learning the occupancy. Many thanks for the question. As stated above, the representation of the 2D binary occupancy is not the same as the NeRF representations. The updating of the occupancy grid is along with the training of NeRF and is not separated.Therefore, we provide the running time of our method in the below table.

We thank the reviewer for the valuable and insightful comments, and the positive comments such as "the first method", "effective" and "I like the idea of modeling occupancy with a MLP". We address the detailed questions and comments below.

Q1: Homogeneous MoE and Heterogeneous MoE. We are very sorry for the misleading caption of Table 4 and the text in the Section 4.5 HMoE. We forget to mention the experiment details of this ablation study. In Table 4, both the Homo. and Heter. MoE are trained with the imbalanced gate loss and without density loss for 40K steps and than their occupancy networks are used to guide the training of Switch-NeRF. The accuracy in Paper Table 4 are the final reconstruction accuracy of the guided stage. These experiments are to show that with our HMoE and imbalanced gate loss, our method can already learn very good occupancy. The Homogeneous MoE with imbalanced gate loss cannot learn reasonable occupancy so if we use its occupancy network to guide the training of Switch-NeRF, many important (occupied) points will be classified as unoccupied. Therefore, the accuracy drops a lot. We have fixed this misleading components in the revised version. In our main experiments, the empty space expert is set as an Identity layer and have been shown as effective across quite different large-scale scenes. We also add experiments of different sizes of the empty space experts in Table 9 in Appendix G and the table below. In the table, Homo. is a large MLP with 7 layers. The 4-layer version can learn better occupancy wile the Identity version can gets results very close to Switch-NeRF. Therefore, from our extensive experiments, with an Identity layer as the expert space expert can learn very good results. The output of the Identity empty space expert will still go through the prediction head so the network can still be trained.

Q2: Guided training. The main motivation of freezing the occupancy network is to decrease the computation in the training of NeRF and to sample denser samples. In the training of occupancy network, we uniformly samples 3D points along rays. And the unoccupied points still need gradient and optimization for training the occupancy which consumes more memory and computation. Since our occupancy network converges very fast, we can freeze the occupancy network to accelerate the classification of 3D points and sampling denser samples to train NeRF methods, without extra gradients and optimization for the occupancy network.

We thank the reviewer for the valuable and insightful comments, and the positive comments such as "solve a very important problem", "novel" and "useful". We address the detailed questions and comments below.

Q1: Intuitions of the scene scales. Many thanks for this question. We describe the large-scale scenes in our experiments to give more intuitions. The Mega-NeRF dataset comprises 5 scenes each consisting of $2k$ to $6k$ images with a resolution from $4k$ to $5k$, covering urban regions from 0.15 to 1.3 $km^2$. The Block-NeRF dataset includes a street scene with $12k$ images at $1k \times 1k$ resolution, covering an Urban street of $1km$ long. These scenes are very challenging due to the huge image data and the coverage of large-scale regions. The experiments have shown that our compact occupancy network can handle these large-scale scenes effectively. From our perspective, for the small scale objects, the efficiency problem is not as critical as the large-scale scene because of the efficient representation such as the hash-encoding, small data volume, dense multi-view images and requirements of small resolution of grids. In the large-scale urban scenes, due to the large data volume and large scene range, the learning of occupancy becomes much harder and critical, and our proposed compact occupancy network can effectively solve these problems. Our newly add experiments on large-scale scnens with Instant-NGP as baseline also show that it can handle different large-scale representations (Appendix C, Table 6 and Reviewer KM5v Q4).

Q2: Visualization in Fig. 1. Many thanks for this suggestion. We add more details about the visualization in the caption of Fig. 1 and the main text.

Q3: Blank image in Fig 4b. Many thanks for the question. This is not a mistake. The bottom right image of Fig 4b shows the unoccupied points with alpha. Points with small alpha will be shown as transparent. Please refer Section 4.2 Occupancy visualization for more details. Since with density loss, our method can predict more accurate occupancy, the points in this image are all with very small density and alpha. Therefore, all of the points of the bottom right image of Fig 4b are transparent when shown with alpha and thus the image is all blank. Compare with the bottom left image, the bottom right image shows the effectiveness of our density loss to separate the occupied and unoccupied points more clearly. We add more description of this image in the caption and main text in the revised version.

Q4: Code releasing. We will surely release our codes upon publication to benefit the community.

We thank the reviewer for the valuable and insightful comments. We address the detailed questions and comments below.

Q1: The difference between our method and Switch-NeRF. We would like to clarify our motivation, novelty, and our distinctive features compared to Switch-NeRF.

In this paper, we solve a fundamentally different problem compared to Switch-NeRF in large-scale NeRF modeling. The Switch-NeRF aims to learn the decomposition of a large-scale scene for scalable training. Our paper targets learning a compact representation of a binary occupancy of a large-scale scene, which can provide effective guidance to learn a large-scale NeRF efficiently. This is the first time that this problem has been introduced for large-scale NeRF modeling.

To tackle this important and new problem of modeling the occupancy of a large-scale scene, our approach is designed based on our insights on the problem, and our contribution is three-fold:

(i) A novel heterogeneous MoE structure with scene experts and a tiny Empty Space Expert. This structure is designed to effectively model the high heterogeneity of occupied and unoccupied 3D scene points.

(ii) A novel and effective gate imbalanced loss. If we train the HMoE with the balanced loss as used in the original Switch-NeRF that targets a homogeneous modeling of the scene, the model cannot learn the occupancy because the scene distribution is highly heterogeneous with a lot more points belonging to the empty space compared to the surface. It is thus very important to enforce a training prior to make the network distinguish the occupied and the unoccupied 3D points. The proposed gate imbalanced loss can effectively control the portion of points dispatched to the empty space expert, which is a critical design for the successful large-scale occupancy learning.

(iii) A novel and effective density loss. Our designed density loss can directly control the gating network according to the predicted density. It enables our occupancy network to explicitly model the important scene geometry prior, which is the density of an unoccupied scene point is much smaller than that of an occupied scene point. The density loss helps to ensure the learning of more accurate scene occupancy.

In the experiments of the main paper, our method has shown consistent advantages of accuracy, converging speed, and the GPU memory overhead over the occupancy grid method and the Switch-NeRF trained with the same time. In our newly added experiments (Table 6 in the appendix and in Q4) with the Instant-NGP as a baseline, our method also outperforms the Instant-NGP by large margins. These experiments show that our method gains consistent performance improvements compared to the occupancy grid method in various datasets and baselines, demonstrating the effectiveness of our proposed approach for learning a compact large-scale scene occupancy.

In conclusion, our paper tackles a fundamentally different problem compared to Switch-NeRF and all our three novel contributions are closely related to our insights on the new problem. Our method confirms consistent advantages over the occupancy grid method.

Q2: Reconstruction quality. We would like to highlight our clear advantages of accuracy, converging speed, and the GPU memory overhead over the occupancy grid method and the Switch-NeRF, which have been extensively verified in our experimental results. For example, our model achieves an improvement of about 0.6 PSNR over Switch-NeRF on Sci-Art (Table 3 in the main paper) when being trained with the same time. We obtain a similar PSNR with only 8 hours training compared to Switch-NeRF using 14 hours (Fig. 9 in the main paper). Our method also uses much less memory (2.4G) than Switch-NeRF (10.5G) and the occupancy grid (5.8G) as reported in Table 3 of the main paper. In our newly added experiments (Table 6 in the appendix and also in Q4 of the rebuttal) with the Instant-NGP as the baseline, our method also outperforms the Instant-NGP by large margins, e.g., 1.07 points of PSNR on Campus and 0.85 on Building. These experiments on the different datasets and baselines demonstrate the effectiveness and the advantages of our proposed approach.