Coordinate-Aware Modulation for Neural Fields

We sincerely appreciate all insightful comments and efforts in reviewing our manuscript. All the reviewers acknowledged the innovative and impactful approach proposed in our paper, supported by extensive experiments. We intend to respond to each of your comments one by one, offering thorough explanations and clarifications as required.

[W1] More baseline (instant-NGP) for image tasks

Following the reviewer's suggestion, we assessed I-NGP for our image tasks. While I-NGP demonstrated impressive results in Natural image regression, highlighting its superiority in overfitting, it fell short in terms of image generalization. In contrast, CAM consistently shows high performance in both tasks, demonstrating its overall effectiveness. We will include this result in the upcoming draft.

[W2-1, Q2] Clarification of the choice for modulation parameters

CAM is applied not based on the signal's property but on the input coordinates. Although video signals contain spatial components, frame-wise methods such as FFNeRV only take time as the input coordinate to represent each corresponding video frame. This has been demonstrated as an efficient solution in terms of accuracy and training/inference speed, as opposed to incorporating both spatial and time coordinates (pixel-wise methods) in video representation literature. In our experiments, we adopted this more practical and applicable baseline and used the input time coordinate to determine the modulation parameters, showing a significant improvement over the baseline. Given that the time coordinate is the only option to determine the modulation parameters in this context, our ablation study presented in Table 6 did not include the video experiment. We would like to claim that this result highlights the wide and general application of CAM.

[W2-2] CAM based on both direction and time.

As described in Section 3.0, our approach involves assigning priority to input coordinates (when input coordinates include various components) for modulation parameters to avoid the curse of dimensionality. The prioritization is arranged in the following order: time, view direction, and space. This specific ordering is based on experimental results from tasks that involve several coordinate components, such as NeRF and dynamic NeRF, with the results detailed in Table 6. Furthermore, we have conducted additional evaluations of CAM regarding both direction and time coordinates in the following table. Although it increases performance compared to the baseline, using a single prioritized component demonstrates the most efficient result. This result will be included in the updated version.

[W3, Q3] Description of the variance in Figure 8

The variance in Figure 8 denotes the mean of the variance of all pixels at the same channel. Formally, the variance $v$ of $H\times W$ pixel-wise $C$-channel features $X \in \mathbb{R}^{C\times H\times W}$ can be expressed as follows,

$v = {1\over C}\sum_{ch=1}^{C} var^{(H,W)}(X_{ch}),$

where $var^{(H,W)}(\cdot):\mathbb{R}^{H\times W} \rightarrow \mathbb{R}$ computes the variance of $H\times W$ values and $X_{ch}\in\mathbb{R}^{H\times W}$ is features at the channel $ch$. We would like to clarify the information in the revised draft.

[W4] Clarification of notation

In our paper, we have described the modulating parameter of MLP's intermediate features as $\gamma(X;\Gamma)$ (and $\beta(X;B)$), where the function $\gamma(\cdot;\Gamma):\mathbb{R}^{N\times D} \rightarrow \mathbb{R}^{N}$ and the input coordinate $X\in\mathbb{R}^{N\times D}$, with the dimension of the input coordinates $D$. However, we overlooked specifying $\gamma_n(X;\Gamma)$ as the scale value for batch $n$ in Equations 1 and 2, even though we precisely mention 'the scalars $\gamma_n(X; \Gamma)$ and $B_n(X; B)$ denote the scale and shift value, respectively, for ray $n$' in Equation 4.

The notation $X^{(\cdot)}$ denotes the subset of input coordinates based on our proposed priority. For example, this could be the view-direction coordinate $X^{(\phi, \theta)}$ in NeRFs or the time coordinate $X^{(t)}$ in dynamic NeRFs. Given that image and video representations involve only spatial and time coordinates, respectively, we use the complete input coordinate by denoting it as $X$. We would like to revise and provide clarity on these aspects in the revised version of our draft.

[Q1] GPU usage of HM

We extend our sincere apologies for the incorrect statement that 'HM decodes at 10fps using a GPU'. Upon the reviewer's correct observation, we realize our misinterpretation of the report by Hu et al.[1] and acknowledge that HM natively supports only CPU usage, barring any specialized implementation. This error would be rectified in the revised version of our draft.

[1] Hu, Z., Xu, D., Lu, G., Jiang, W., Wang, W., & Liu, S. (2022). FVC: An end-to-end framework towards deep video compression in feature space. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4), 4569-4585.

We sincerely appreciate all insightful comments and efforts in reviewing our manuscript. All the reviewers acknowledged the innovative and impactful approach proposed in our paper, supported by extensive experiments. We intend to respond to each of your comments one by one, offering thorough explanations and clarifications as required.

[W1] More informative figure We appreciate the reviewer's thoughtful comment on our main figure. We revised Figure 1, as shown in Link. We would like to include this figure in the updated draft.

[W2] Brief description of the functional form $\Gamma$ and $B$

$\Gamma$ and $B$ are grid structures to represent the scale and shift factors, where each grid is trained as a function of coordinates with infinite resolution, outputting coordinate-corresponding components. The output in infinite resolution is aggregated by nearby features in the grid based on the distance between the coordinates of the input and neighboring features. We would like to include this in the appendix of the upcoming draft.

We sincerely appreciate all insightful comments and efforts in reviewing our manuscript. All the reviewers acknowledged the innovative and impactful approach proposed in our paper, supported by extensive experiments. We intend to respond to your comment, offering thorough explanations and clarifications as required.

[W1, Q1] Baseline model with our own implementation

For our experiments, we utilized the official codes for all baseline methods, including FFNeRV, NerfAcc, Mip-NeRF, and Mip-NeRF 360, and reported the values in the original papers. The only exception was FFN, which was originally developed in Jax a few years back. Due to its older environment, we opted for a Pytorch implementation to simplify the experimental process. We followed all the original settings of FFN, and successfully reproduced the reported value (1.9 PSNR higher for the Natural dataset and 0.4 lower for the Text dataset). It's important to note that, apart from the integration of CAM, all other configurations in our experiment were exactly the same. This consistency underscores the effectiveness of CAM in a clear and unbiased manner. If we understand your question correctly, we would like to include the reported values in the original paper and clarify this.

We sincerely appreciate all insightful comments and efforts in reviewing our manuscript. All the reviewers acknowledged the innovative and impactful approach proposed in our paper, supported by extensive experiments. We intend to respond to each of your comments one by one, offering thorough explanations and clarifications as required.

[W1] More references

We are grateful for the reviewer's insightful suggestion and would like to include the mentioned references for a more thorough discussion in our work.

[W2] Inference speed and memory

We report the inference speed and GPU memory requirements of the MLP-based method (NerfAcc), grid-based method (K-Planes), and CAM, evaluated on the 'Mic' scene. K-Planes requires small memory while showing slow inference. CAM reduces the original NerfAcc's speed when testing chunk size is small. However, increasing the testing chunk size reduces the speed gap between using CAM and not using it. Intriguingly, CAM even lowers memory usage under these conditions. We interpret that CAM facilitates a more effectively trained occupancy grid and helps bypass volume sampling, offsetting the additional computational demands introduced by CAM itself.

[W3] Ablation on the choice for modulation parameters

As described in Section 3.0, our approach involves assigning priority to input coordinates for modulation parameters to avoid the curse of dimensionality. The prioritization is arranged in the following order: time, view direction, and space. This specific ordering is based on experimental results from tasks that involve several coordinate components, such as NeRF and dynamic NeRF, with the results detailed in Table 6. Furthermore, we have conducted additional evaluations of CAM regarding both direction and time coordinates in the following table. Although it increases performance compared to the baseline, using a single prioritized component demonstrates the most efficient result.

[W4] CAM on tri-plane representation

When applied CAM to a tri-plane method (K-Planes), there is no significant improvement observed (from 34.10 to 34.12 in the 'Mic' scene). This is because general grid-based methods avoid spectral bias by utilizing grids with high resolution and large channels, where the decoder MLP primarily serves to aggregate these grid features. This is the reason for the substantial memory demands of grid-based methods. In contrast, CAM represents a novel approach to mitigate spectral bias, achieving this with notably compact parameters.

[W5] Application for DreamFusion

We have applied CAM to the official DreamFusion codebase, observing some impact on the outcome, as shown in Figure Link. However, due to time limitations, we were unable to conduct a more in-depth exploration. Exploring this area further, we believe, would constitute an interesting direction for future research.