Mask-Based Modeling for Neural Radiance Fields

Dear Reviewer GXVJ,

Thank you for your comments. Please see below our response to your concerns.

Weakness 1: Do different masking strategies prefer different levels of masking?

Answer 1: In our experiments, we find that the combination of using ray-level as well as view-level masking operation brings the most significant improvements, no matter for our final proposal or for the other three masking variants (i.e., RGB mask and two feature masks). We reckon that masking along the two dimensions together will better exploit the inner correlations among points and across views, in which case the 3D scene prior knowledge will be learnt more adequately. We already search for the optimal masking level (i.e., masking ratio) for our final proposal as well as the three masking variants in the ablation study, and report the best for each in Table 4, which shows the superiority of our final proposal.

Weakness 2: It would be great if more generalizable NeRF models could be integrated with MRVM for comparison.

Answer 2: We add our mask-based modeling strategy to another well-known generalizable NeRF baseline GNT [1], under the cross-scene generalization setting, using LLFF and NeRF Synthetic as evaluation sets. The quantitative comparisons are shown below:

Our proposed MRVM proves to be effective in improving the generalizability for GNT as well.

[1] Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang. Is attention all that nerf needs?

The reviewer thanks the authors for their responses. After reading all the other reviewers’ comments, I would like to keep my rating to endorse this paper for acceptance.

Dear Reviewer y3Pz,

Thank you for your comments. Please see below our response to your concerns.

Weakness 1: Are there any architecture modifications for NeuRay?

Answer 1: We do not make any architecture modifications for NeuRay. Naturally, there exist inherent correlations among the sampled points and across the reference views for generalizable NeRFs, regardless of different network architectures. Our mask-based pretraining target is to well-exploit such correlations for improving NeRF's generalizability. Even for the MLP-based NeRFs, such correlations, as the 3D scene prior knowledge, can still be learnt through our proposed MRVM and encapsulated in the implicit MLP networks as well. That's because MLP networks also have some kind of inner correlations among different input signals. Our experiments demonstrate the applicability of our proposed MRVM for both MLP-based and Transformer-based backbones.

Weakness 2: An additional ablation experiment is needed to validate the effectiveness of the BYOL-like module.

Answer 2: Thanks for your valuable suggestion, we add the ablation as the reviewer suggested and refer as NeRFormer+MRVM (w/o BYOL) in the table below:

It validates the effectiveness of the BYOL-like module in our final proposal. Intuitively, as many previous contrastive-learning methods [1,2] pointed out, the usage of projector and predictor avoid the model collapsing into a trivial solution, that is, the two branches learn similar but meaningless representations, which impairs the effectiveness of pretraining.

[1] Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning.

[2] Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton A Simple Framework for Contrastive Learning of Visual Representations.

Question 1: It is better to provide more details about the architecture of the used backbones in the paper.

Answer 3: Thanks for your valuable suggestion, we'll provide more details about the architectures of the used backbones in our revision to make them more clear. We'll put them in the revised appendix.

Dear Reviewer chDn,

Thank you for your comments. We'll release the code and models after acceptance. Please see below our response to your concerns.

Weakness 1: Whether the proposed method will cause double parameters and inference costs?

Answer 1: Most of the Neural Radiance Field works adopt coarse-to-fine sampling strategy, using two independent network modules handling sampled points at coarse and fine stage respectively. We reuse the two network module branches as target and online branch, aiming to introduce the fewest additional parameters. Therefore, the only additional parameters introduced by our MRVM are the light-weight projector and predictor, which brings negligible additional computational overhead (only +0.2% parameters compared to the baseline model). Please refer to Table 4 for more details. Moreover, since the projector and predictor are discarded at inference time, the inference speed is consistent with the original baseline methods.

Weakness 2: The terminology in method section is not easy to follow. Besides, what is the parameter updating procedure for other network parts apart from the projector?

Answer 2: We use the terminology 'projector' and 'predictor' following the previous work BYOL [1]. The projector represents the neural network projecting the intermediate feature to another latent space and the predictor represents the neural network predicting target representations from online ones. As shown in Figure 1, since the gradient is stopped manually for the target projector, it could only be updated using moving average from the online counterpart. Noting that we still adopt vanilla NeRF's rendering loss $L_{nerf}$ for supervision during our mask-based pretraining stage, as shown in Equation 10. Therefore, the rest of other neural network modules are updated using gradient-decent algorithm as before. Thanks for your valuable suggestion, we'll make the statements more clearly in our revision.

[1] Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning.

Weakness 3: Suggestions on illustrating the masking strategies in ablation study with more details in the main paper.

Answer 3: Thanks for your valuable suggestion. Due to the page limits, we are only able to describe the three masking variants briefly in the main paper, we'll try to make them more precise in our revision.

Dear Reviewer Jc3W,

Thank you for your comments. Please see below our response to your concerns.

Weakness 1: Can EMA and masking be used without the other rather than tested together?

Answer 1: We'd like to firstly explain our approach of using EMA and masking together. As we mentioned in Section 3.2, apart from the vanilla rendering loss, we use an additional feature-alignment loss $L_{mrvm}$ in latent space following the successful formulation introduced by BYOL [1]. This requires the gradient to be manually stopped at the target (coarse) branch, as we illustrated in Figure 1. Therefore the projector in target branch can not be updated with gradient decent algorithm, that's why we use EMA to update its parameters using Equation 5. If only using masking without EMA, as we understand, is to make a copy of the parameters of the projector from online branch to target branch every iteration. We find this will make the mask-based pretraining stage instable based on our experiments $-$ the loss term $L_{mrvm}$ fluctuates and is not easy to converge.

[1] Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning.

Weakness 2: Shouldn't the masking prediction be focused at the more important (close-to-surface) high density locations, rather than at the coarse sampling?

Answer 2: As the reviewer understands, the extra fine samples are taken, as in the classical NeRF, according to an initial estimation of density from the coarse samples. We agree that the close-to-surface points sampled at fine stage are more important, but we do not treat them as prediction target due to the fact that : 1) We can not predict the latent embeddings of these points directly from the coarse branch since they are newly sampled at fine stage and do not exist at the coarse stage. That makes it impractical to treat coarse branch as online and fine branch as target. 2) We do perform an ablation study of making a copy of the overall fine-branch to serve as target and update via moving average, dubbed as Feat mask^2^ in Section 4.3. However we find this variant, with more additional parameters, exhibits inferior performance compared to our final proposal. Please refer to Table 4 for quantitative comparisons and refer to the Discussion part in Section 3.2 for more detailed analysis.

Weakness 3: Impact on training time is not clear.

Answer 3: The generalizable NeRF adopts the training strategy of cross-scene pretraining followed by per-scene finetuning. It is firstly pretrained across various scenes. After pretraining, the model can either be tested directly on a novel scene or further finetuned on that specific scene. Our proposed Masked Ray and View Modeling serves as an auxiliary task when performing cross-scene pretraining, which requires an additional finetuning stage without masking to maintain the consistency between training and testing. Overall it takes about 30% additional training time compared to the baselines.

Weakness 4: Some minor suggestions.

Answer 4: The blue dashed arrows represent the message-passing along sampled points on the ray. Thanks for your valuable suggestion, we'll include such notations in our revision.

Question 1: Have you tried comparing to per-scene NeRFs?

Answer 5: The per-scene finetuning results as shown in Table 3 have already be comparable to several per-scene NeRF methods, and it shrinks the gaps compared to SOTA performance on per-scene NeRFs.

Question 2: Could you provide an ablation showing the contribution of applying only one of EMA / Masking?

Answer 6: The ablation for different mask-based pretraining strategies are presented in Table 4. We find only applying masking will make the mask-based pretraining instable. As mentioned in Answer 1, if copying the parameters from online projector to target ones every iteration, we observe fluctuation for the loss term $L_{mrvm}$, making it not easy to converge in our experiments.

We do not try to only apply EMA without masking since we propose a mask-then-predict pretraining task, simply abandoning masking operation is irrelevant to our proposal. Besides, the rest of the network parameters except the target projector are updated using normal gradient-decent algorithm, with no need for the EMA parameter updating strategy.

Question 3: How sensitive is the choice of $\lambda$, the weight of the mrvm-loss?

Answer 7: Experimentally we find $\lambda$ suitable between 0.1~0.5.

Dear Reviewer fmpu,

Thank you for your comments. Please see below our response to your concerns.

Weakness 1: It might be interesting to investigate the few-shot results in more detail, like the way GNT and GNT-MOVE does.

Answer 1: Thanks for your valuable suggestion. We add our proposed Masked Ray and View Modeling to a well-known generalizable NeRF baseline GNT [1], and compare under the same few-shot settings as the reviewer mentioned:

The numerical numbers are PSNR, SSIM and LPIPS from left to right respectively. It shows our proposed MRVＭ still benefits under the few-shot settings.

[1] Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang. Is attention all that nerf needs?

Question 1: A clarification for the details of the way in performing masking operation.

Answer 2: As the reviewer understands, we use the same randomly sampled input points to both coarse and fine branches, in addition to the additional importance sampled inputs used by the fine branch. However, we perform randomly masking operation to all the points input into the fine branch, not just to the additional importance sampled ones. Please refer to Section 3.2 for more details.

Question 2: Have any other masking strategies rather than random masking been tried out?

Answer 3: This paper proposes a general framework for incorporating mask-based pretraining strategy into the NeRF community. We only use random masking to prove its effectiveness. Thanks for the reviewer's suggestions, we believe more specially-designed masking choices are worth exploring in the future works.

Dear Reviewer P3FF,

Thank you for your comments. Please see below our response to your concerns.

Weakness 1: Why using masked-based modeling to learn high-level global information and what are the advantages and necessities?

Answer 1: In our Abstract, Introduction and Method 3.1 Section, we have emphasized the motivation and benefits of using mask-based modeling for improving the generalizability of Neural Radiance Field. A mask-then-predict pretraining objective has been widely applied in MLM [1] and MIM [2] field for learning a high-level global representation. As in NeRFs, we reckon that the learnt 3D scene global information, which contains the correlations among sampled points and across reference views, endowing the model with better capacity to effectively generalize to novel scenes with limited observations.

[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.

[2] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners.

Weakness 2: The authors should add the masked modeling design to more generalizable NeRF models to demonstrate its wide applicability.

Answer 2: We add our mask-based modeling strategy to another well-known generalizable NeRF baseline GNT [3], under the cross-scene generalization setting, using LLFF and NeRF Synthetic as evaluation sets. The quantitative comparisons are shown below:

Our proposed MRVM proves to be effective in improving the generalizability for GNT as well.

[3] Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang. Is attention all that nerf needs?

Besides, we'd like to clarify that FreeNeRF and SparseNeRF as the reviewer mentioned are not generalizable NeRF methods which may not be compared to our proposed approach targeting on generalizable NeRFs directly. These two methods focus on solving the few-shot reconstruction problem especially designed for a single particular scene, while generalizable NeRF methods use a shared network to represent various different scenes.

Question 1: Can this method only handle object-centric and facing-forward scenes?

Answer 3: We currently perform experiments on the object-centric and forward-facing scenes because the previous generalizable NeRF baselines are mainly evaluated on these two kinds of scenes. For more complicated scenes like in-the-wild unbounded 360 degree scenes as in the tank-and-temple dataset, or the large-scale complex indoor scenes, most SOTA methods in the NeRF community still learn an independent NeRF per scene for satisfying rendering quality. However, we see no obstacles that may hinder our proposed method in applying for other types of scenes, and we'll leave these as a future work.

Thanks for the author’s detailed responses. I have read the response and other reviewers’ feedback. The authors address my doubt and so I raise my score to accept.

Dear Reviewer s7zj:

Thank you for your comments. Please see below our response to your concerns.

Weakness 1: The computational overhead is not well discussed in the paper.

Answer 1: Most of the Neural Radiance Field works adopt coarse-to-fine sampling strategy, using two independent network modules handling sampled points at coarse and fine stage respectively. We reuse the two network module branches as target and online branch, aiming to introduce the fewest additional parameters. Therefore, the only additional parameters introduced by our MRVM are the light-weight projector and predictor, which brings negligible additional computational overhead (only +0.2% parameters compared to the baseline model). Please refer to Table 4 for more details. Moreover, since the projector and predictor are discarded at inference time, the inference speed is consistent with the original baseline methods.

Weakness 2: The author may need to clarify the role of mask-based pretraining more clearly in the paper.

Answer 2: Yes we agree with the reviewer's comment. Thanks for the valuable suggestion and we will clarify this point in our revision.

Question 1: Can the authors provide more details on the computational efficiency of MRVM-NeRF?

Answer 3: Please refer to Answer 1.

Question 2: Can the $Pred^f$ function take additional inputs besides the latent feature like geometrical information?

Answer 4: Theoretically, $Pred^f$ can take additional geometrical inputs like position $x$ and view direction $d$ by modifying the predictor network structure. It's worth exploring whether this operation will bring greater performance gain in the future work. Actually, the geometrical information has already been introduced at the beginning of the generalizable NeRF framework. In Section 3.1, the input latent embedding $h_i^j$ has already contained the information of position $x$ and direction $d$ $-$ in the way that $x$ and $d$ are projected into a latent embedding after positional encoding, which are then added to $f_j^j$ , $c_i^j$ to produce input vector $h_i^j$. We apologize for the lack of these details and will clarify them in our revision.