Refined Tensorial Radiance Field: Harnessing coordinate based networks for novel view synthesis from sparse inputs

Qualitative results on dynamic NeRFs : All the qualitative and visual results for dynamic NeRFs are based on the scenario with 25 views. In this analysis, even though the proposed method exhibits some foggy artifacts between frames, we note that when dealing with 25 views, learning time-dependent information becomes a challenging task in terms of training perspective. When compared to full views, we are only utilizing a maximum of 25% of the full dataset, and in terms of time information, the data provides coarser details about time. This creates an extremely challenging situation. Nevertheless, the proposed method does not produce floating artifacts or break the continuum structure, but also this model consistently tracks objects over time. As a result, the key point is that the proposed method remarkably enhances performance across all scenes, particularly for scenes like "bouncingballs" and "standup."

The number of parameters and training time : The proposed method does take longer compared to fast training methods like iNGP and TensoRF. However, it's worth noting that our approach still maintains reasonable training times. As shown in Table H.10, we've demonstrated that early stopping effectively preserves performance across all baselines, including our own, and all training procedures are completed around 30 minutes.

We conduct a comparison between TensoRF, K-Planes, and our model on the static NeRF dataset, limiting the training steps to 0.15 million and 8 views used for training. Rendering time is evaluated using 200 frames from the test dataset. We also experiment with less parameterized models. The numbers in brackets indicate the channel count in the multi-plane features each axis. We note that TensoRF(20) encountered training instability issues and failed to train on the scenes {chair, ficus, mic}, denoted by a hyphen. In the K-Planes model, featuring multi-resolution multi-plane characteristics, the total number of channel dimensions is the product of the number of resolutions and the channel dimension at each resolution.

The following table shows the experiment result for dynamic NeRF, including the number of parameters, average training time and rendering time. We constrain the training steps to 0.15 million and 25 views used for training. Rendering time is evaluated using 20 frames from the test dataset.

• We include the results of iNGP in the manuscript

An intriguing point to note is that when we reduce the number of parameters in the baselines to match ours, such as K-Plane, and TensoRF, their performance suffers, and they sometimes fail to converge. However, when we halve the number of parameters, our model's reported performance only experiences a slight decrease while still outperforming the baselines, and it also achieves slightly faster training and rendering speeds.

Regarding rendering time, our method may not be the most efficient. However, when combined with recent rendering techniques employed in iNGP, it can significantly accelerate rendering speeds. Nevertheless, we believe that this paper's primary focus is on robust training and criticizing the side-effects of existing regularization on explicit representation. Therefore, the rendering pipeline is not the main concern here, and this issue does not align with the core message of the paper.

Regarding dynamic NeRFs, the training and rendering times do not exhibit a significant gap compared to static NeRFs. We will provide more details on this result shortly.

Thank you for thoughtful comments.

Contribution of this paper : This paper emphasizes that relying solely on explicit representation has limitations when dealing with sparse data. Interpolation-based plane features struggle to train reliably without an adequate amount of data points. Moreover, we found that a high level of denoising regularization can lead to color distortion, as evident in the case of K-Plane in Fig 5 and Fig E.2, and TensoRF often fails to converge in some instances. In contrast, our proposed method consistently represents the training datasets without artificial distortion. Even when excessive denoising regularization affects performance, as seen in the visual outputs in Fig 5 and Fig E.2, our approach avoids depicting artificial distortion, although the rendered images appear blurry. Therefore, we consistently demonstrate that the proposed method enhances performance and training stability through the integration of coordinate networks and multi-plane encoding. While our approach is straightforward and a combination of existing techniques, in this context, we believe it offers a valuable new axis for advancing NeRFs and improving their applicability.

Examining characteristics and analyzing explicit representation : We conducted an ablation study to understand the role of each representation, and this content is included in Appendix F. In essence, our proposed method leverages the inherent properties of coordinate networks without sinusoidal encoding, which biases towards low-frequency details. During training, an increase in the engagement of multi-plane features does not lead to floating artifacts.

The figures in Figure G.6 were not produced by separate trained models; they were all generated by our proposed method once it converged. After training, we masked all multi-plane features and named them coordinate networks. We found that coordinate networks are responsible for low-frequency details. As we increase the engagement of multi-plane features, the rendered images become finer and finer. This experiment solidly supports our claim that coordinate features handle low-frequency details while multi-plane features are responsible for finer details.

Theoretical aspects of the proposed method :

We acknowledge that theoretical aspects hold promise for future research directions. The significance of our experimental results is clear: existing explicit representations often overlook inappropriate embeddings in sparse regimes, and denoising regularization can introduce side effects like learning instability or color distortion. Focusing on the core empirical findings and contributions can help ensure that the message is clear and impactful to target readers.

We believe that the proposed approach can be linked to k-NN methods, which take into account global priors. A line of k-NN works has tackled hyper-parameter selection using adaptive methods by considering global patterns in relation to local details [1]. Similar to the original NeRF [2] and Fourier feature [3], we plan to explore this aspect further in the context of k-NN domains in future research.

[1]Anava, Oren, and Kfir Levy. "k*-nearest neighbors: From global to local." Advances in neural information processing systems 29 (2016).

[2]Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1 (2021): 99-106.

[3]Tancik, Matthew, et al. "Fourier features let networks learn high frequency functions in low dimensional domains." Advances in Neural Information Processing Systems 33 (2020): 7537-7547.

Sinusodial encoding in the proposed method :

To clarify, the proposed method does not employ positioning encoding for coordinates. In the static NeRFs, the input is represented as $x \in \mathbb{R}^3$ for the coordinate network, while in the dynamic NeRFs, it is $x \in \mathbb{R}^4$. Positioning encoding is specifically used for ray direction. Additionally, for the case of ficus, we utilize positioning encoding for appearance features to avoid artificial color distortions. However, apart from this instance, the proposed method does not rely on positioning encoding for coordinates. The detailed information is described in Fig 4 and the attached configuration files.

Thank you for your thoughtful comments.

Novelty and Contribution : We've observed that when dealing with sparse inputs, relying solely on explicit representation tends to lead to overfitting the training data. While the regularization method mentioned by reviewer YmuZ does show some effectiveness, it's important to note that baseline models perform better under specific hyperparameters and scenarios. However, they often struggle to converge in other cases and can generate artifacts that appear like real data but are not part of the training set.

In few-shot scenarios, the significance of stable training is widely recognized, even if the proposed techniques are not entirely fresh. We demonstrated that the previous methods still struggle to achieve consistent performance with sparse inputs as they tend to overly emphasize local details due to linear interpolation between feature grids rather than considering the low-frequency details. Even though we identify that classical approaches such as early-stopping and fewer capacity models are not effective in this setting, consequently, they cannot eliminate entirely all artifacts.

More precisely, as shown in Table 1, numerical results reveal that K-Planes exhibit greater scene-to-scene variance compared to the proposed method. Qualitatively, as demonstrated in Fig 5 and Figure E.12, K-Planes exhibit color-related issues due to denoising regularization. Particularly, as the denoising regularization value increases, color distortion becomes more pronounced. TensoRF experiences instability problems when dealing with sparse inputs, as seen in Table 3 and Table E.6, failing to converge when $\lambda_1$ exceeds 0.01. Stability and consistent performance across different scenarios are crucial, especially in few-shot training settings, where achieving the best performance in a few cases may not generalize well to other experiments.

On the other hand, Our proposed method consistently achieves top performance and robustly trains NeRFs, making it a reliable choice for various experimental cases. Furthermore, it outperforms other baselines in dynamic NeRF scenarios, indicating its stability and applicability in few-shot regimes. We have updated the literature survey to contain these arguments.

Additional supplementary animations : We have uploaded as many supplementary videos as possible, including cases with varying denoising regularization.

Thanks for your thoughtful comments.

Contribution to NeRFs in the sparse inputs : In few-shot scenarios, the significance of stable training is widely recognized, even if the proposed techniques are not entirely fresh. We demonstrated that the previous methods still struggle to achieve consistent performance with sparse inputs as they tend to overly emphasize local details due to linear interpolation between feature grids rather than considering the low-frequency details. Even though we identify that classical approaches such as early-stopping and fewer capacity models are ineffective in this setting, they cannot eliminate all artifacts.

Our proposal suggests addressing both low-frequency and high-frequency details in sparse-input situations to avoid overfitting to only the finest details during training. In this context, our approach argues that a combination of multi-plane encoding and coordinate networks not only successfully captures both local and global patterns of the target signals but also provides a robust training method using channel-wise curriculum learning, emphasizing a global perspective before delving into finer details.

Theoretical aspects : Focusing on the core empirical findings and contributions can help ensure that the message is clear and impactful to target readers. Therefore, it is entirely reasonable to consider excluding theoretical parts because they are out of the scope of this paper.

While acknowledging that theoretical aspects hold promise for future research directions, our experimental results are crucial, so we want to emphasize our core message: existing explicit representations often overlook inappropriate embeddings in sparse regimes, and denoising regularization can introduce side effects like learning instability or color distortion.

We believe that the proposed approach might be seen as akin to k-NN methods, taking into account global priors. A line of k-NN works have tackled hyper-parameter selection using adaptive methods [1]. Similar to the original NeRF [2] and Fourier feature [3], we plan to explore this aspect further in the context of k-NN domains in future research.

[1]Anava, Oren, and Kfir Levy. "k*-nearest neighbors: From global to local." Advances in neural information processing systems 29 (2016).

[2]Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1 (2021): 99-106.

[3]Tancik, Matthew, et al. "Fourier features let networks learn high frequency functions in low dimensional domains." Advances in Neural Information Processing Systems 33 (2020): 7537-7547.

Explanation for Figure 3 : We agree with reviewer D6bb. The Figure 3 serves as a conceptual illustration of our method. The encoder mentioned in the Figure 4 encompasses the components of all layers in the Figure 3. Specifically, the 'MLP layer' refers to the linear layer preceding the hidden-layer in the Figure 4, while 'residual MLP' includes concatenation and the layers following the hidden layer. To clarify, the MLP denotes the coordinate networks, which are solely responsible for handling the coordinates. For example, $x \in \mathbb{R}^3$ in the case of static NeRFs, and $x \in \mathbb{R}^4$ in the case of dynamic NeRFs.

However, we note that the images in Figure 3 were not produced by separate trained models; they were all generated by our proposed method once it converged. After training, we masked all multi-plane features and named them coordinate networks.

The number of parameters and training time : The proposed method does take longer compared to fast training methods like iNGP and TensoRF. However, it's worth noting that our approach still maintains reasonable training times. As shown in Table H.10, we've demonstrated that early stopping effectively preserves performance across all baselines, including our own, and all training procedures are completed around 30 minutes.

We conduct a comparison between TensoRF, K-Planes, and our model on the static NeRF dataset, limiting the training steps to 0.15 million and 8 views used for training. Rendering time is evaluated using 200 frames from the test dataset. We also experiment with less parameterized models. The numbers in brackets indicate the channel count in the multi-plane features each axis. We note that TensoRF(20) encountered training instability issues and failed to train on the scenes {chair, ficus, mic}, denoted by a hyphen. In the K-Planes model, featuring multi-resolution multi-plane characteristics, the total number of channel dimensions is the product of the number of resolutions and the channel dimension at each resolution.

The following table shows the experiment result for dynamic NeRF, including the number of parameters, average training time and rendering time. We constrain the training steps to 0.15 million and 25 views used for training. Rendering time is evaluated using 20 frames from the test dataset.

• We include the results of iNGP in the manuscript

An intriguing point to note is that when we reduce the number of parameters in the baselines to match ours, such as K-Plane, and TensoRF, their performance suffers, and they sometimes fail to converge. However, when we halve the number of parameters, our model's reported performance only experiences a slight decrease while still outperforming the baselines, and it also achieves slightly faster training and rendering speeds.

Regarding rendering time, our method may not be the most efficient. However, when combined with recent rendering techniques employed in iNGP, it can significantly accelerate rendering speeds. Nevertheless, we believe that this paper's primary focus is on robust training and criticizing the side-effects of existing regularization on explicit representation. Therefore, the rendering pipeline is not the main concern here, and this issue does not align with the core message of the paper.

Regarding dynamic NeRFs, the training and rendering times do not exhibit a significant gap compared to static NeRFs. We will provide more details on this result shortly.

Minor modification : We revised the manuscript the Equation 3 to account for the disappearance of overlapped regions.

The number of parameters and training time : The proposed method does take longer compared to fast training methods like iNGP and TensoRF. However, it's worth noting that our approach still maintains reasonable training times. As shown in Table H.10, we've demonstrated that early stopping effectively preserves performance across all baselines, including our own, and all training procedures are completed in around 30 minutes.

We conduct a comparison between TensoRF, K-Planes, and our model on the static NeRF dataset, limiting the training steps to 0.15 million and 8 views used for training. Rendering time is evaluated using 200 frames from the test dataset. We also experiment with less parameterized models. The numbers in brackets indicate the channel count in the multi-plane features each axis. We note that TensoRF(20) encountered training instability issues and failed to train on the scenes {chair, ficus, mic}, denoted by a hyphen. In the K-Planes model, featuring multi-resolution multi-plane characteristics, the total number of channel dimensions is the product of the number of resolutions and the channel dimension at each resolution.

The following table shows the experiment result for dynamic NeRF, including the number of parameters, average training time and rendering time. We constrain the training steps to 0.15 million and 25 views used for training. Rendering time is evaluated using 20 frames from the test dataset.

• We include the results of iNGP in the manuscript

An intriguing point to note is that when we reduce the number of parameters in the baselines to match ours, such as K-Plane, and TensoRF, their performance suffers, and they sometimes fail to converge. However, when we halve the number of parameters, our model's reported performance only experiences a slight decrease while still outperforming the baselines, and it also achieves slightly faster training and rendering speeds.

Regarding rendering time, our method may not be the most efficient. However, when combined with recent rendering techniques employed in iNGP, it can significantly accelerate rendering speeds. Nevertheless, we believe that this paper's primary focus is on robust training and criticizing the side-effects of existing regularization on explicit representation. Therefore, the rendering pipeline is not the main concern here, and this issue does not align with the core message of the paper.

Regarding dynamic NeRFs, the training and rendering times do not exhibit a significant gap compared to static NeRFs. We will provide more details on this result shortly.

Thank you for your thoughtful comments.

Contribution to NeRFs in the sparse inputs (Literature survey) : This paper highlights that denoising regularization techniques, which have been effective on explicit representation, struggle to reliably train NeRFs in various situations despite previous research [1,2]. While some baselines may perform better under specific hyperparameters and scenarios, they often fail to converge in other cases and produce artifacts that look real data but are not part of the training set.

As shown in Table 1, numerical results reveal that K-Planes exhibit greater scene-to-scene variance compared to the proposed method. Qualitatively, as demonstrated in Fig 5 and Figure E.12, K-Planes exhibit color-related issues due to denoising regularization. Particularly, as the denoising regularization value increases, color distortion becomes more pronounced. TensoRF experiences instability problems when dealing with sparse inputs, as seen in Table 3 and Table E.6, failing to converge when $\lambda_1$ exceeds 0.01. Stability and consistent performance across different scenarios are crucial, especially in few-shot training settings, where achieving the best performance in a few cases may not generalize well to other experiments.

On the other hand, the proposed method consistently achieves top performance and robustly trains NeRFs, making it a reliable choice for various experimental cases. In few-shot scenarios, the significance of stable training is widely recognized, even if the proposed techniques are not entirely fresh. While we identify that classical approaches such as early-stopping and fewer capacity models are ineffective in this setting, they cannot eliminate all artifacts, it outperforms other baselines in both static and dynamic NeRF scenarios, indicating its stability and applicability in few-shot regimes. We have updated the literature survey to contain these arguments.

[1] Fridovich-Keil, Sara, et al. "K-planes: Explicit radiance fields in space, time, and appearance." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Cao, Ang, and Justin Johnson. "Hexplane: A fast representation for dynamic scenes." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Real-world dataset : We evaluate the proposed method on the real-world Tanks and Temples dataset, where it was compared with the baseline TensoRF models, including a regularization ($\lambda_1 = 0.001$). We selected the 'Tanks and Temples' dataset because it contains scenes where the camera distribution has objects-wise, rather than scenes with forward-facing camera distribution. As stated in the literature survey in the manuscript, training on object-facing data is more challenging when dealing with sparse input situations.

In this table, we present SSIM of test dataset produced by the proposed method and baselines. All methods are trained on 10% of training views.

The proposed method consistently achieves higher SSIM scores across all scenes, indicating its superior capability in preserving the structural integrity and overall composition of scenes.

While PSNR is a valuable metric for image quality, it can be biased due to the lack of mask information for white backgrounds and the inclusion of full-resolution ($1920\times1980$) in the tanks and temples dataset. We show the PSNR result as follows.

In the revised manuscript, we have included both quantitative and qualitative results while varying the number of training views. Please refer to the section I in the appendix of the revised manuscript.

To sum up, the proposed method distinguishes itself from the baselines through its robust ability to preserve the global context of scenes, handle sparse input data effectively, and render images that are visually realistic. The experimental results consistently demonstrates how the proposed method enhances its potential for broader application in real-world scenarios, where input data is often sparse and incomplete.