MutualNeRF: Improve the Performance of NeRF under Limited Samples with Mutual Information Theory

We extend our gratitude to the reviewer for the comments and suggestions. Below, we address the primary concerns that have been raised.

Q1: The proposed method introduces more complex training requirements and utilizes a large model, CLIP, for assessing semantic space distance, which may impact training time and peak memory consumption.

A1: We have explored alternative semantic metrics to quantify "image similarity," such as DINOv2[1] and MAE[2], as encoders in our loss computation. Below are the comparative results with NeRF on the Blender dataset:

Our findings indicate that different encoders exhibit similar performance improvements, demonstrating that our framework is generalizable and not heavily rely on clip to calculate semantics.

Q2: Figure 4 shows that some regions, such as the stairs, are not improved. Why might this be the case? Could this indicate limitations in the Mutual Information design? Specifically, which types of regions can be enhanced by the macro and micro losses? Currently, it is unclear which specific issues the proposed framework addresses. Could the authors provide a more detailed explanation?

A2: Thank you for pointing out the limitation shown in Figure 4. The observed lack of improvement in certain regions, such as the stairs, might be due to insufficient feature alignment between the synthetic and reference views in these areas. This issue may arise because such regions often contain intricate geometric or high-frequency texture details that are challenging for the mutual information-based design to fully capture, particularly when the macro and micro features have lower correspondence.

Our proposed framework leverages macro and micro mutual information losses to enhance overall structure and fine-grained detail consistency, respectively. The macro loss is effective for improving large-scale structural features by maximizing high-level correspondence, whereas the micro loss helps to align local fine details, which might be missing or mismatched across views. We plan to investigate improved methods for enhancing intricate regions, such as adaptive feature extraction or incorporating geometry-aware constraints to better align these areas.

Q3: Does the proposed macro loss also improve the performance of 3DGS-based methods? If so, did the authors experiment with sparse-view 3DGS splatting methods?

A3: Our paper primarily focuses on NeRF synthesis with limited training data. Due to the limited time for the rebuttal, we are unable to directly conduct experiments related to 3DGS. We believe extending this approach to 3DGS is a valuable direction for future work.

Q4: What would happen if we applied CLIP as a perceptual loss to support FreeNeRF training?

A4: Applying CLIP as a perceptual loss to support FreeNeRF training could potentially improve the semantic consistency between the generated and real images, as CLIP's feature space is trained to align well with human perception. We think incorporating CLIP as a perceptual loss might introduce additional computational complexity, as observed with our integration of CLIP in the NeRF framework. In future experiments, we plan to further explore the application of CLIP to FreeNeRF to assess its effectiveness in improving the synthesis quality under sparse-view settings.

We thank the reviewer once again for the valuable and helpful suggestions.

References

[1] Oquab M, Darcet T, Moutakanni T, et al. Dinov2: Learning robust visual features without supervision[J]. arXiv preprint arXiv:2304.07193, 2023.

[2] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.

We greatly appreciate the reviewer's comments and valuable suggestions.

Q1: This methodology introduces significant complexity, especially in sparse view sampling, involving greedy algorithms and complex mutual information metrics. However, the observed improvements over simpler baselines are relatively minor, which may not justify the added complexity.

A1: We appreciate the reviewer's feedback. The primary goal of this paper is to provide a novel perspective on the problem of sampling under limited data. We identified specific application scenarios within two NeRF setups and demonstrated performance improvements over baseline algorithms in both cases. Additionally, more application scenarios can be explored in future work. We followed the settings of ActiveNeRF. Since we only need to compute CLIP similarity and camera distance, the time overhead is only 1/5 of that of ActiveNeRF.

Q2: The attempt to address the task of few-shot novel view synthesis through minimization of mutual information between viewpoints have already been explored, especially in CVRP 2022 paper InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering. The methodology of this paper seems to be re-interpretation of ideas used in DietNeRF (semantic space) and InfoNeRF (pixel space) within the perspective of mutual information theory. I ask the authors to give a more thorough theoretical comparison between this work and the methods that I have mentioned.

A2: We thank the reviewer's comment. First, we provide two application scenarios: sparse view sampling and few-shot synthesis. For the second scenario, our approach is entirely different from that in the InfoNeRF[1]. InfoNeRF uses Shannon Entropy to define the entropy of a discrete ray density function, while we base our method on concepts from mutual information, which involve significant conceptual differences.

References [1] Kim, Mijeong, Seonguk Seo, and Bohyung Han. "Infonerf: Ray entropy minimization for few-shot neural volume rendering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

We thank the reviewer for the comments and constructive suggestions. In the following, we address the main concern raised. Please find the details below.

Q1: How about using mutual information for 3DGS?

A1: Our paper primarily focuses on NeRF synthesis with limited training data. Due to the limited time for the rebuttal, we are unable to directly conduct experiments related to 3DGS. We believe extending the idea of mutual information to 3DGS is a valuable direction for future work.

Q2: What is the rendering speed of the proposed method, especially when compared with 3DGS?

A2: We did not add this loss at every iteration. In fact, applying it every 100 iterations yields very good results, with the time overhead being less than 1.2 times.

Q3: Why choose to pick images instead of training as a whole? Are there images that do not belong to the target scene?

A3: The issue of limited acquisition budget is well addressed in ActiveNeRF[4] presented at ECCV 2022. The main idea of their paper is that NeRF usually requires a large number of posed images and generalize poorly with limited inputs. And it takes a whole observation in the scene to train a well-generalized NeRF. This poses challenges under real-world applications such as robot localization and mapping, where capturing training data can be costly, and perception of the entire scene is required.

Our approach strictly follows the acknowledged setup in the domains of active learning and NeRF, ensuring a fair comparison with state-of-the-art (SOTA) methods.

Finally, we thank the reviewer once again for the effort in providing us with valuable suggestions. We will continue to provide clarifications if the reviewer has any further questions.

References

[1] Yu, Alex, et al. "pixelnerf: Neural radiance fields from one or few images." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[2] Yang, Yifan, et al. "Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Liu, Tianqi, et al. "MVSGaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo." European Conference on Computer Vision. Springer, Cham, 2025.

[4] Pan X, Lai Z, Song S, et al. Activenerf: Learning where to see with uncertainty estimation[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 230-246.

Thanks again for your valuable feedback! Could you please let us know whether your concerns have been addressed? We are happy to make further updates if you have any other questions or suggestions.

We express our gratitude to the reviewer for the insightful comments. We address the main concerns below. If the reviewer believes there are additional issues with our paper that have not been addressed, we are very willing to continue the discussion.

Q1: The motivation for pixel space distance is not clarified. According to Definition 3, the pixel space distance is the expectation of distance between any two points of rays. The authors should clarify the motivation behind Definition 3 since it is important for the following parts. The semantic space distance is fairly reasonable.

A1: We appreciate the reviewer's feedback. Our intuition is mainly based on the idea that capturing light rays from different positions of an object provides richer information about it. Since the object varies continuously internally, we measure the richness of the information provided by evaluating the differences at every point along two light rays.

Q2: The relationship between mutual information and few-shot view synthesis is not clear.

A2: This paper primarily focuses on providing a new perspective for understanding sampling under limited data. Sparse view sampling and few-shot synthesis are just two application scenarios we explored. More application scenarios can be provided in future work.

Q3: For the few-shot view synthesis, the proposed method needs to evaluate the semantic distance between randomly rendered images and ground truth images. Therefore, this will bring additional training costs. The analysis should be provided.

A3: We did not add this loss at every iteration. In fact, applying it every 100 iterations yields very good results, with the time overhead being less than 1.2 times.

Q4: Compared with NeRF, 3D Gaussian splatting is much better in both training and rendering. The authors should conduct experiments based on 3DGS.

A4: Our paper primarily focuses on NeRF synthesis with limited training data. Due to the limited time for the rebuttal, we are unable to directly conduct experiments related to 3DGS. We believe extending the idea of mutual information to 3DGS is a valuable direction for future work.

We would be delighted to address any further inquiries you may have. Please feel free to reach out with any additional questions or concerns!