ResFields: Residual Neural Fields for Spatiotemporal Signals

We would like to thank the reviewer for considering our work as exceptionally clearly described and thoroughly evaluated and ablated, as well as for the insightful comments and questions regarding the proposed method and its performance. In the following, we address the remaining discussion points related to the capacity of the low-rank representation and the potential computational overhead of the proposed framework.

1. The capacity of the low-rank representation for modeling dynamic components of varying complexity.

We especially appreciate the suggestion and we will incorporate an additional study to benchmark the learning capabilities of the low-rank weights for signals with multiple diverse dynamic segments.

Learning signals with many independent parts imposes additional challenges for both ResFields and coordinate MLPs in general. The coordinate MLP weights possess the ability to compress a spatio-temporal signal into a compact representation, mainly because the source signal has a significant amount of repetitive information which is effectively represented by the optimized network weights. The shared weights of the ResFields have the same property, as they tend to store information that is common across the entire signal, whereas the low-rank weights accommodate for topological and dynamic changes for effective learning.

We conduct the following experiment to analyze the learning capacity of our low-rank weights in the case of dynamic scenes with varying complexity. We create three videos with different levels of difficulty: 1) A corner case when every frame depicts a novel scene (a video composed from the first 250 images from the DIV2K dataset), 2) the bikes video containing 6 segments, and 3) the cat video containing only 1 segment. All of these three videos are trimmed and cropped to the same length (250 frames) and resolution ( 351x510 ).

Then, we perform the 2D video approximation (analogous to Sec. 4.1.) by learning the mapping from a space-time coordinate to RGB color (10% of pixels are not seen during the training and are left for evaluation).

Generally, we observe that increasing the number of independent video segments indeed has a significant effect on the model performance. This can be attributed to the lack of information that can be shared across the entire signal and efficiently encoded in the base matrix weights. However, despite this, our method successfully improves quality over the vanilla Siren, even in the most challenging case of a random video (19.4 vs 29.18 PSNR for our model with rank 40). This improvement can be attributed to (a) the increased capacity of the pipeline thanks to residual modeling and (b) the ability of the neural network to discover small patterns that could be effectively compressed into shared weights even in the extreme case of a video with completely random frames. The latter behavior is especially apparent in the simplest case when ResFields uses only rank 1 (19.4 vs 23.34 PSNR for the random video sequence). Please also note that increasing the capacity in our case does not lead to significant computational overhead, as we demonstrate in the main submission and further emphasize in the next section.

Dear Reviewer KYty,

We would like to kindly inquire whether you have had the opportunity to review our updated manuscript, along with the responses addressing your raised discussion points. Your feedback is highly valuable to us, and we welcome any additional questions or comments you may have.

Best regards,

Authors

We would like to thank the reviewer for considering our work effective and easy to use, as well as for the insightful comments and references. In the following, we do our best to put the presented ResFields approach in the context of the provided related methods. We provide the quantitative evaluation when possible, and elaborate on the conceptual differences and limitations when a straightforward comparison is not feasible.

Comparisons to flow-supervised NeRF methods [1,2]

Please note that both methods [1,2] require optical flow supervision, which in practice is hard to obtain for scenes considered in the paper. The Owlii sequences exhibit a magnitude of uniformly colored texture patterns that has a significant impact on the optical-flow estimators (e.g. RAFT).

Unfortunately, [1] has no published code at the present day, significantly limiting our ability to provide quantitative evaluation against the approach in a timely manner.

We have evaluated DynIBR [2] on the Owlii benchmark (Tab. 3). However, the method fully fails on the dataset with large camera view angle differences. This behavior can be partially attributed to the reliance of the method on the synthesized depth maps obtained from the off-the-shelf monocular depth regressors, which produce inconsistent results across largely varying camera views. Additionally, DynIBR being an image-based rendering technique in its nature, also exhibits a rather limited novel-view synthesis and geometry reconstruction capability and serves more as a video stabilization technique. Finally, training DynIBR (and its variation with ResFields) is not computationally feasible for us as the original method already requires 8 A100 GPUs for training.

However, in the following we provide strong evidence that ResFields approach generally benefits the task of modeling motion fields, the fundamental building block of the aforementioned flow-based NeRF methods [1,2].

Specifically, DynIBR [2] uses a 256-neuron ReLU-MLP to model bi-directional motion fields by predicting coefficients of the DCT bases, while Deformable NeRF [1] uses a deformation network of Nerfies (128-neuron ReLU-MLP) to predict motion as SE(3). In both cases, the NeRF models are explicitly supervised with optical flow to learn motion/scene flow.

To demonstrate the benefit of ResFields for both of these motion fields, we use 4 sequences of DeformingThings from Sec. 4.2, and learn bi-directional motion fields. We use 80% of tracked mesh vertices to learn the motion field and the remaining 20% for evaluation. As a supervision, we use L1 error between the predicted and the ground truth motion.

The table below reports the L1 error (scaled by x10^3) for the evaluation points. Additionally, we have included one additional modeling option from [A], where the motion field is simply predicted as an offset vector.

Note that ResFields greatly benefits learning motion fields across all settings. Moreover, the particular architecture from DynIBR (DCT with 256 neurons) is less powerful than its much smaller and faster counterpart (128 neurons) with ResFields for both forward (2.18 vs 2.47) and backward (2.19 vs 2.48) motion while being around three times faster (16.5 vs 5.6 training it/s).

References:

[A]: Prokudin et. al., “Dynamic Point Fields” ICCV 2023

The rebuttal addresses my concerns well. I will raise the score.

We thank the reviewer for considering our work simple and effective, and that it can be a good contribution to the community. In the following, we address the remaining discussion points:

1. Alternative weight matrix factorization, key intuition.

We have explored diverse modeling and factorization techniques (Tab. 5, 6) based on prior work, including different tensor factorizations (CP, Tucker), however, we have not considered the proposed example. We will gladly include it in our ablation study (Tab. 6).

The goal of our parametrization is to achieve high learning capacity while retaining good generalization properties. To achieve this for a limited budget in terms of the number of parameters, we want to allocate as few as possible independent parameters per time step (represented by the vector $v(t)$) and as many globally shared parameters $M_i$. Allocating more capacity towards the shared weights will enable 1) the increased capacity of the model due to its ability to discover small patterns that could be effectively compressed in shared weights and 2) stronger generalization as the model is aware of the whole sequence through the shared weights.

Given these two objectives, we have designed $v(t)$ to have very few parameters (R) and $M_i$ to have the most parameters (RxNxM).

In light of this intuition, the factorization proposed by the reviewer appears to be suboptimal, as it generally allocates more capacity in the per-frame independent part $v(t)$ (NxR), and less into the shared weights $M_i$ (RxM).

We further verify this intuition quantitatively and show that the alternative design has a lower learning capacity, due to a larger number of independent parameters that need to be re-learned for every time step; for the same reason, this parametrization is more prone to overfitting. Below, we provide the extended Tab. 6 on the video approximation task:

Quantitative results confirm that our parametrization achieves higher learning capacity (higher training PSNR) and stronger generalization (higher test PSNR) while using fewer parameters.

Specifically, our parametrization with the rank of 10 (8.7M parameters) compared to the proposed example with the rank of 40 (19.3M parameters) demonstrates:

• test PSNR: 39.59 vs 36.67
• train PSNR: 40.8 vs 39.01
• the number of parameters: 8.7M vs 19.3M

We will include these results and further elaborate on our intuition in the revisited manuscript and the supplementary material.

Dear Reviewer ZKNQ,

We would like to kindly inquire whether you have had the opportunity to review our updated manuscript, along with the responses addressing your raised discussion points. Your feedback is highly valuable to us, and we welcome any additional questions or comments you may have.

Best regards, Authors