GML-NeRF: Gate-guided Mutual Learning Framework for Neural Rendering

We thank the reviewer for the constructive feedback and thoughtful comments. We address the detailed questions and comments below.

R4-Q1: Demonstration of Figure 4 and CV balance regularization

Thank the reviewer for the questions. We give a more detailed description of Figure 4 and CV balanced regularization below to clarify how GML-NeRF conducts the CV balanced regularization.

(1) In Figure 4, the visualizations of gating scores in three scenes depict the specialties of one sub-NeRF (sub-NeRF 1) for two different regions (e.g., the body of the truck and other regions), instead of the weights of two sub-NeRFs.

(2) We clarify that the CV balanced regularization is not to enforce equal weighting between sub-NeRFs. The CV loss is calculated on the average gating score within a batch rather than being applied to each ray individually. In this way, it helps to prevent the gate module from collapsing onto a specific sub-NeRF on all rays while maintaining sub-NeRFs' specialties. We highlight this mechanism in Section 4.4 in the revision.

R4-Q2: Discussion of soft gate module

Thank the reviewer for the insightful question. As clarified above, the CV balanced regularization is to prevent the gate module from collapsing onto a specific sub-NeRF and would not interfere with the learning of the gate module. The visual representation in Figure 4 reinforces this point by illustrating how the gate module learns a reasonable parameter allocation across different scenes. To further support this point, we conduct an ablation study of CV balanced regularization on the TAT dataset and add the results to the Appendix D. First of all, the rendering quality degrades without CV balanced loss, proving its necessity. Then, we apply the CV regularization only for the first half of the training time and find that the performance is comparable to that of GML-NeRF.

R4-Q3: Performance of GML-NeRF

Thank the reviewer for the consideration of performance. Overall, compared to Instant-NGP baseline, our scheme has improvements in the PSNR metric across various datasets: 1.02 on mask-TAT, 0.98 on TAT, 0.57 on 360v2, 0.49 on free dataset and 0.82 on ScanNet dataset. All the improvements are achieved by the default configuration of two sub-NeRFs. Notably, as the number of sub-NeRFs increases, an additional 0.4 PSNR gain is observed on the ScanNet dataset. We also check the performance of recently published and related work, such as F2-NeRF, which also implements a multi-NeRF framework on Instant-NGP. F2-NeRF claims the PSNR metric improvements of 0.15 on 360v2 and 1.91 on free dataset, which is on par with GML-NeRF effect. Additionally, on the 360v2 dataset, ZipNeRF obtains 0.74 PSNR gain compared to MipNeRF360, and 3D Gaussian Splatting-7k obtains 0.3 PSNR gain compared to INGP.

Furthermore, GML-NeRF goes beyond comparison with existing SOTA single-NeRF methods, extending to the comparison with other multi-NeRF training frameworks based on NGP. In these comprehensive evaluations, GML-NeRF shows superiority over other multi-NeRF training methods.

Finally, GML-NeRF acts as a multi-NeRF training method, which is adaptable to various types of scenes and compatible with different single-NeRF frameworks. Extensive experiments across complex scenes presented in the main paper affirm its adaptable ability. Additionally, we implement the GML-ZipNeRF framework, which is a combined version of GML-NeRF and ZipNeRF (SOTA single-NeRF method). The experimental results prove GML-NeRF's effectiveness and potential for integration with different frameworks, as detailed in the response to reviewer PcY6 (R2-Q1) and included in the Appendix.

R4-Q4: Details of uniform fusion in Table 2 For the uniform fusion, we remove the soft gate module and just average sub-NeRFs' outputs in both training and inference time. Without the gating mechanism introduced by the gate module, sub-NeRFs lose the ray specialties, resulting in performance degradation.

Dear Reviewer Vfdc,

Thanks once again for your valuable time in reviewing our paper. Did our previous responses address your concerns satisfactorily? If you have any further questions, please kindly inform us. We are more than willing to provide further clarifications.

Authors

Dear reviewer Vfdc,

Thank you once more for the review. We eagerly anticipate your feedback on our response. Did our previous response and revision address all your concerns? If you have any further questions, please let us know and we will try our best to provide additional clarification before the DDL.

Authors

We thank the reviewer for the constructive feedback and thoughtful comments. We address the detailed questions and comments below.

R3-Q1: Analysis of the performance improvement

Thank the reviewer for the insightful question. We conduct an ablation study in the response to reviewer MyQ8 (R1-Q3), and the results highlight the effect of independent MLP decoders rather than feature grid. Within the hybrid representation, the explicit part (feature grid) encodes the features of 3D spatial points, while the implicit representation encodes the ray information. As our motivation is to consider the different visibility of a target region to different views, using multiple MLP encoders with different ray specialties is reasonable and effective. Although the number of learnable parameters hardly increases, GML-NeRF achieves a more optimal capacity allocation in the ray dimension, which helps to increase the model's generalization ability. Additionally, as different rays may pass through the same region of 3D space, weight sharing for the feature grid helps to facilitate training.

R3-Q2: Details and discussion of Block-NGP

For a fair comparison, the structure of the model in Block-NGP is the same as the one employed in GML-NeRF. Despite implementing the same model structure, Block-NGP does not obtain performance improvements compared to the baseline (Instant-NGP). It's essential to clarify that this lack of improvement is not attributed to the model's structural design. As mentioned in response to R3-Q1, the pivotal design in GML-NeRF lies in the use of multiple MLP encoders with different ray specialties rather than the feature grid. Such assertion is also supported by the ablation studies addressed in response to reviewer MyQ8 (R1-Q3).

The reason behind Block-NGP's poor results primarily stems from three aspects. Firstly, the manual image allocation scheme based on image position is not suitable for the scenes captured with arbitrary trajectories. Despite implementing the allocation through the K-means cluster algorithm, the allocation result may not be reasonable. Secondly, compared to the discrete 3D points, different rays have complex spatial relationships, such as crossing or passing through the same region, which makes Block-NGP's "hard" allocation ineffective. Thirdly, Block-NGP implements a view-level gating module, outputting a score to each image, which is a coarser granularity compared to pixel-level fusion.

Furthermore, in the inference time, Block-NGP doesn't perform the interpolation of results from sub-NeRFs. This implementation is due to the inherent difficulty in defining a proper interpolation method when dealing with various complex scenes captured by free trajectories.

R3-Q3: Implementations of ablation studies

Thank the reviewer for pointing out the confusion in Tables 3 & 4. We appreciate the opportunity to provide clarification on the confusion.

(1) Ablation Study of Fusion Dimension (Table 3): In this ablation study, we explore the superiority of GMl-NeRF over the point-based multi-NeRF method. The point-based method adopts a model structure similar to GML-NeRF, incorporating a shared feature grid and a soft gate module. The difference is that the soft gate module encodes the 3D points' data and outputs a gating score for each point. The fusion process is completed on the point dimension, followed by volume rendering.

(2) Ablation Study of Fusion Granularity (Table 4): In this ablation study, both experiments are conducted within the GML-NeRF framework. For the image-level fusion, we directly average the ray directions of all pixels in each image to obtain the image directions. Then, the gate module outputs a score to each image, which is a coarser granularity compared to pixel-level fusion.

Dear Reviewer 3Q2w,

Thanks once again for your valuable time in reviewing our paper. Did our previous responses address your concerns satisfactorily? If you have any further questions, please kindly inform us. We are more than willing to provide further clarifications.

Authors

Dear reviewer 3Q2w,

Thank you once more for the review. We eagerly anticipate your feedback on our response. Did our previous response and revision address all your concerns? If you have any further questions, please let us know and we will try our best to provide additional clarification before the DDL.

Authors

R2-Q4: Demonstration of scalability study and the responding figure

Thank the reviewer for pointing out several issues and questions regarding Figure 5. The feedback is important for refining the paper, and we would like to address the concerns below.

(1) Scalability results clarification: We acknowledge the omission that we only report the scalability results on scene 0046 of the ScanNet dataset rather than the average PSNR result of four scenes. We have modified it in the revision.

(2) Hidden dimension scaling for the blue curve: The blue curve scales the hidden dim of both the geometry MLP and the color MLP while maintaining the resolution of the hash feature grid.

(3) GML-NeRF with two independent sub-NeRFs: This implementation is different from the one of GML-NeRF, in which the difference is whether the feature grid is shared among the sub-NeRFs. The number of parameters increases significantly within the design of unshared feature grids. We have slightly modified this figure in the revision to clarify this detail.

We hope that our modification and clarification can help!

R2-Q5: Effect of scaling up the resolution of feature grid

Following the reviewer's suggestion, we further conduct experiments to explore the impact of increasing the number of parameters in the feature grid, and the results are incorporated into Figure-5 in the latest revision. While maintaining default values for resolution and the number of hash levels, we opted to increase the hash table size to scale up the feature grid, consistent with the test methodology employed in the original NGP paper. As the results show, although the number of parameters increases significantly, the performance gains achieved by increasing the "log2_hash_table_size" from 19 to 24 are limited, which is consistent with the results in the original paper.

R2-Q6: G(r) in Equ-3

G(r) is the K-dimensional vector representing the prediction confidence of each sub-NeRF for the ray r. Since we insert a softmax normalization layer at the end of the gate module, the elements of the G(r) sum to one.

We thank the reviewer for the constructive feedback and thoughtful comments. We address the detailed questions and comments below.

R2-Q1: Results of ZipNeRF and Gaussian Splatting

Thank the reviewer for the advice regarding the comparison of GML-NeRF with the latest baselines.

(1) We implement a ZipNeRF version of GML-NeRF, named GML-ZipNeRF, and evaluate the performance on the 360v2 dataset. The training settings are kept the same as the original paper, including the training iterations and batch size. We add the comparison results in the Appendix G, showing that integrated with GML-NeRF, ZipNeRF can also obtain performance gains.

(2) Due to time constraints, we could not verify the effect of GML-NeRF upon the Gaussian Splatting. A possible implementation to integrate GML-NeRF and 3D Gaussian Splatting is constructing an ensemble of 3D Gaussian representations for independent rendering. Subsequent gate-guided fusion can be conducted in a similar way to this article. We leave it for future exploration.

As a multi-NeRF training framework, GML-NeRF is essentially orthogonal to the structure and training method of single-NeRF. In our work, for the benefit of training efficiency and its wide application, we build and validate GML-NeRF upon the Instant-NGP. Nevertheless, it can be integrated with other single-NeRF frameworks, including ZipNeRF and Gaussian Splatting, and so on.

R2-Q2: Consideration of training efficiency

Thank the reviewer for the important reminder. Considering the importance of training efficiency on the practicability of the NeRF training frameworks, we mainly choose the baselines with high training efficiency (NGP, DVGO, F2-NeRF, etc.) and other multi-NeRF training methods for comparison, while providing the results of other baselines as the extra reference (NeRF, MipNeRF360,...). The consideration for training efficiency is also the motivation for adopting hybrid representations in GML-NeRF. Using the hybrid representation, ensemble size growth only increases the number of MLPs, which brings only a moderate parameter size increase. We highlight this consideration in the structure design part of the method section in the revision.

Additionally, although our GML-NeRF implemented upon Instant-NGP may not exhibit comparable performance to MipNeRF360 trained with more than one day, it acts as a flexible multi-NeRF training framework with the potential to be integrated with various single-NeRF frameworks (NeRF, MipNeRF, etc). As the above experimental results show, implemented upon SOTA single-NeRF method ZipNeRF, an NGP-version of MipNeRF360, GML-ZipNeRF shows better rendering quality, which further proves the effectiveness of the proposed method.

R2-Q3: Results of more sub-NeRFs

We adopt the k=2 setting by default mainly for the sake of experiment convenience, given its lower memory footprint in training. Nevertheless, GML-NeRF can consistently obtain performance gains as the number of sub-NeRFs increases, as shown in Figure-5.

To provide a more comprehensive demonstration of this point, we provide per-scene results of GML-NeRF with more sub-NeRFs on the ScanNet dataset in the Appendix F. As the results show, GML-NeRF with 4x sub-NeRFs performs better than the one with 2x sub-NeRFs across all scenes.

Dear Reviewer PcY6,

Thanks once again for your valuable time in reviewing our paper. Did our previous responses address your concerns satisfactorily? If you have any further questions, please kindly inform us. We are more than willing to provide further clarifications.

Authors

Dear reviewer PcY6,

Thank you once more for the review. We eagerly anticipate your feedback on our response. Did our previous response and revision address all your concerns? If you have any further questions, please let us know and we will try our best to provide additional clarification before the DDL.

Authors

R1-Q3: Effect of shared feature grid

Thank the reviewer for the questions. In GML-NeRF, the shared feature grid is a better choice than the unshared one. We conduct an ablation study on the TAT dataset and add the results to the Appendix D. The experiment results show that the model employing a shared feature grid outperforms its counterpart with multiple independent feature grids. Such superiority comes from two aspects.

Firstly, within the hybrid representation, the feature grid is responsible for encoding features of 3D spatial points, while the MLP encoder is designed to encode view information. The crucial design of independent MLP decoders aligns with our visibility-aware motivation, thereby enhancing the view-dependent effect. Secondly, the training complexity will also increase as the trainable parameters increase. With the limited amount of training data, increasing the number of feature grids leads to poor convergence. By contrast, as different rays may pass through the same region of 3D space, weight sharing for the feature grid helps to facilitate training.

Although the model with unshared feature grids can still achieve satisfactory results, particularly in terms of the SSIM and LPIPS metrics, we still recommend using the shared feature grid scheme due to its parameter efficiency.

R1-Q4: Limitation of GML-NeRF on translucent objects

Thank the reviewer for the insightful suggestion.

We think one limitation of GML-NeRF is that compared with single-NeRF or multi-NeRF (with hard gating) methods, it will introduce additional inference latency, although achieving better rendering performance. When the ensemble size is large, we might need to choose several sub-NeRFs instead of using all sub-NeRFs to reduce the inference latency. In addition, in order to strike a good balance between the rendering performance and inference latency, we might need to investigate how to decide the best choice of ensemble size (i.e., $K$) for a given scene.

As for the handling of translucent objects. In our view, we think the difficulty of tackling is a problem faced by all NeRF methods, and GML-NeRF doesn't bring extra difficulty. We'd like to know if the reviewer thinks that GML-NeRF has introduced extra in rendering transparent objects than other work and look forward to further suggestions.

We thank the reviewer for the constructive feedback and thoughtful comments. We address the detailed questions and comments below.

R1-Q1: Mechanism of depth regularization

Thank the reviewer for the questions. To empirically analyze the effect of depth mutual learning, we conduct an ablation study on the TAT dataset and add the results to the Appendix.D following the reviewer's suggestion. When using the average of the sub-depths as the target depth (all sub-NeRFs have equal regularization strength), the rendering quality will be slightly worse (PSNR 21.419 vs 21.708). This observation highlights the pivotal role of gate-guided depth mutual learning. In the GML-NeRF framework, an ensemble of sub-NeRFs is jointly optimized via the soft gate module, a crucial component enhancing rendering consistency. Within the design of the soft gate module, all the sub-NeRFs contribute to the same ray, necessitating accurate encoding of scene geometry. Meanwhile, the sub-NeRF with a lower gating score exhibits a relatively poor specialty for the ray, resulting in the inaccuracy of geometry encoding. A larger regularization force could have a better effect in such cases. By differently penalizing sub-depths based on their respective gating scores, our approach significantly contributes to accurate scene representation and improved rendering quality.

R1-Q2: Analysis of training complexity

Thank the reviewer for the thoughtful questions. For the convenience of the experiments (lower memory footprint in training), we adopt the k=2 setting by default. However, it is crucial to note that GML-NeRF consistently obtains performance gains as the number of sub-NeRFs increases, as shown in Figure-5. We add per-scene results in the Appendix F for reference. Moreover, the training difficulty will not increase significantly with the increase in the number of sub-NeRFs. In our experiments, we find that the model with four sub-NeRFs converges faster than the one with two sub-NeRFs while achieving better rendering quality with the same training iterations. The convergence curves are also added to the Appendix F. The ease of training convergence can be attributed to two aspects. On the one hand, the feature grid is shared among multi-NeRFs, and thus, the number of learnable parameters increases marginally. On the other hand, as the neural network is better at fitting low-frequency information, our gate module (a 4-layer MLP without sinusoidal position encoding) has implicitly incorporated "smoothness prior", leading to closer rays to be more possibly assigned closer gating scores. Such smoothness prior is similar to what the reviewer mentioned, helping to reduce the training complexity with more sub-NeRFs.

Dear Reviewer MyQ8,

Thanks once again for your valuable time in reviewing our paper. Did our previous responses address your concerns satisfactorily? If you have any further questions, please kindly inform us. We are more than willing to provide further clarifications.

Authors

Dear reviewer MyQ8,

Thank you once more for the review. We eagerly anticipate your feedback on our response. Did our previous response and revision address all your concerns? If you have any further questions, please let us know and we will try our best to provide additional clarification before the DDL.

Authors