How Learnable Grids Recover Fine Detail in Low Dimensions: A Neural Tangent Kernel Analysis of Multigrid Parametric Encodings

We appreciate the reviewers feedback and thank them for their time. We apologize for any confusion in the grid. The grid structure itself is not learnable and is instead set by the hyperparamters L and d (now h to avoid conflict with the input dimensions). At each intersection point of the grid there are learnable parameters that are trained using gradient descent and backpropagation. The structure of this grid is shown in figure 1 and examples of these learned parameters are shown in Appendix C. We consider both a uniform grid of decreasing cells as well as a logarithmic grid in our evaluations. The baseline is a similar point of confusion as it is just the same as the identity encoding. We address this in the question answer below.

The takeaway from this analysis is that the MPE often outperforms the FFE and should be considered in more applications. As pointed out by reviewer pL4m, the MPE has been mainly used in the graphics community. Our work addresses the theoretical underpinning of the MPE in hopes that it sees a wider adoption in areas such as scientific computing. The MPE is often easier to use and does not suffer from the alias problems found in the FFE (Tancik et al., 2020). The MPEs main limitation is the memory requirement for the grid, but this has already been addressed by sparse (Hadadan et al., 2021) and hashgrid (Muller et al., 2022) representation that reduce memory requirements without changing our analysis. The MPE also is not suited for high dimensional problems as the scaling is poor; however, these domains often find the MPE unnecessary because the spectral bias problem is not present. This is why you will see the FFE used as the positional encoding for Transformers but not the MPE.

We present evaluations on a random 100 image subset of ImageNet to show the generalization of our analysis on a wide variety of images. This is shown in Table 1 and Figure 5. The spectra are averaged across each evaluation and show that the MPE consistently outperforms the FFE and network without any encoding. We have also expanded our results with 3D implicit surface representations on meshes from the Stanford graphics dataset and show that the MPE still produces the highest eigenvalue spectrum overall. Muller et al., 2022 has already shown the broad applicability of the MPE. Our work provides the theoretical underpinning for why the encoding works so well.

We appreciate the reviewer’s feedback in strengthening our paper. As we work on a new draft, are there any shortcomings that can be addressed to improve your view of our contributions? Our paper provides valuable insight into how the MPE functions, and it is our hope that this understanding will motivate future work in further optimizing the encoding as well as extend the MPE’s adoption beyond the graphics community.

Questions:

1. Would it be useful for comparison against other image compression algorithms?

We appreciate the reviewer's suggestion but believe that such a comparison would detract from the goal of the original paper. We are using the 2D image regression problem to illustrate the performance of the different results and provide provide empirical evaluations for the theory. We are not necessarily trying to find the most optimal image compression algorithm. We point the reviewer to Tancik et al., 2020 and Muller et al., 2022 for the broad applicability of these encodings.

2. What does identity correspond to in the plots?

Identity and baseline both refer to a network without any encoding. The change in naming is confusing and we have remedied this in the current draft by referring to this network as the baseline throughout. We also made this explicit at the start of section 5.

3. It would be nice to see the actual learned grid superimposed on the image itself.

Figure 1 shows the structure of the grid overlayed on an example image; however, this grid is much finer in practice. The structure of the grid itself is set with hyperparameters before training. Appendix C shows what the learned grid parameters look like at the end of the training. If there are additional figures that the reviewer would find helpful, we would be happy to add them in a future draft.

Hello. We wanted to see if you have been able to read through our responses as well as the updated draft. We would be happy to address any other concerns and would appreciate any additional feedback.

We would like to point the reviewer to recent responses to reviewer EUYS as well as updates to Section 3 in the paper, which explain the connection between the NTK and the learned features in greater detail. The other concerns and questions in your original review have been carefully addressed, including providing expanded evaluations in a more difficult 3D implicit surface domain. Furthermore, we believe that the latest manuscript is much stronger and better aligned with the ICLR audience's expectations. We thank the reviewer for providing feedback allowing us to improve the impact of our work. If you agree that these points have been addressed and there are no further concerns or questions, we kindly ask that you consider raising our paper's rating.

We hope this message finds you well. As the rebuttal deadline approaches, we kindly ask that you share any additional feedback or clarifications on your comments. Your input is invaluable in helping us address concerns effectively and enhance the strength and impact of our work.

We greatly appreciate the time spent reviewing and providing feedback for our paper. We are working to address the notation issues in a new draft. To address the weaknesses, we have included a short description highlighting why the low dimensionality is important. The notation used for the kernel regressions and encodings was chosen to mirror previous works (Tancik et al., 2020; Wang et al., 2020), but we will change the x for kernel regression to be capitalized to represent the entire dataset (as defined in the notation section). Thank you for pointing out the lack of clarity in the original derivation description. The FFE's NTK was well established in Tancik et al., 2020, but the equations were given in a different form that makes it difficult to compare directly to our results. Our derivation does not change their original analysis, but it allows us to more easily compare the structure of the two kernels. We've updated the sentence at the start of section 4 to highlight this distinction. We also thank the reviewer for highlighting the limitations with PSNR. Our results follow the evaluations presented in Tancik et al, 2020, Hadadan et al., 2021, and Muller et al., 2022 which use PSNR. We will reevaluate the results using these new metrics and update a future draft. To address the generalization, we have completed additional evaluations on a 3D implicit surface problem using OccupancyNet (Mescheder et al., 2018) on meshes from the Stanford graphics dataset.

Thank you for your time and feedback. While we update the draft, are there additional details that you would like to see included?

Questions:

1. What is d in figure 1?

The d in Figure 1 refers to the depth parameter of the grid, which describes the number of learnable parameters at each grid corner. As you point out, however, it seems that this name is overloaded. To address this, we have update the value to h in the current draft. Table 1, Table 2, and Figure 1 have been updated to reflect this. We thank the reviewer for helping to approach the notation used in the paper.

2. Is it possible to quantify the improvement of the MPE over the MLP with no encoding?

Sadly, there is no readily available way to quantify the improvement of the MPE spectrum other than providing a lower bound. To address this, we included Figure 2. Though out of scope for the current work, this finding provides an efficient way for future work to optimize or design new parametric encodings as the kernel only needs to be evaluated on the learnable grid. The minimum eigenvalue could then be used during a hyperparameter search or to compare different grid interpolation schemes. Though we are not able to find an exact quantity for this gap, the method laid out is widely applicable to evaluation of parametric encodings.

3. Why isn't everyone using MPE?

This is a good question and was a major motivation for our work. MPEs have strong performance and do not suffer from the aliasing problem found with FFEs (Tancik et al., 2020). One limitation is the amount of memory needed to store the grid, but this problem has been largely addressed by a sparse grid representation (Hadadan et al., 2021) and a hashgrid representation (Muller et al., 2022). It's important to note that these only affect the storage and do not change the analysis presented in this paper. The MPE has been largely published in the graphics community, but we hope to see more papers using MPE in other communities. We also note that there has been some use of the MPE in scientific machine learning applications (Huang and Alkhalifah, 2023), and we hope that our work will improve adoption by providing a more theoretical basis for the MPE, which was already explored for the FFE.

We thank the reviewer for their time and the important feedback. We address your questions below, but would first like to respond to the weaknesses presented. We agree that the neural tangent kernel is a well established method; however, we see this as a strength of our work rather than a detriment to it. Previous work (Jacot et al., 2020; Tancik et al., 2020; Wang et al., 2020; Yang and Salman, 2019) has established that the NTK is representative of real networks in use. The novelty of our work is in the application of the NTK to the multigrid parametric encoding. This analysis provides invaluable insight into the function of the encoding, allowing for better fine-tuning and broader application in the machine learning community. The establishment of the NTK in the literature also lends itself to the generalization of our results on ImageNet. Our evaluation on a wide variety of images shows that the spectrum holds. Previous work by Tancik et al., 2020, Muller et al., 2022, and Hadadan et al. 2021 have shown that the encodings work well on a wide variety of applications, but the eigenvalue spectrum for the MPE was never established. To strengthen our results, we have evaluated the NTK on OccupanyNet (Mescheder et al., 2018) to show that the spectrum holds for implicit surface representations in 3D. Meshes from the Stanford graphics dataset, as used by Tancik et al., 2020, show that the MPE produces a larger eigenvalue spectrum in the 3D case as well. We are preparing these plots for an upcoming draft. Reviewer pL4m corroborates this stance that the ImageNet evaluations are sufficient. The draft also updates the notation in equations 9, 12, and 14 to make the derivative more explicit as they are with respect to the model parameters.

We again thank the reviewer for their time. Are there any other aspects of the paper that we can address to strengthen your impression of the work? We believe that our analysis is important for the wider adoption of the MPE and better use of encodings in machine learning applications.

Questions:

1. Can the theory be applied to other architectures like CNNs?

Yes, the theory can be applied to most common architectures as shown by Yang and Salman, 2019. Our analysis however is purposely restricted to MLPs as described at the start of the introduction. Often there are applications in graphics or scientific machine learning where the network needs to be both small and fast to evaluate. There is also typically not regular structure that can be applied to CNNs, so MLPs are used.

2. Do the presented figures show results only for single-layer, fully connected networks?

We apologize for the confusion. The derivation in section 4 focuses on a single layer as extension to multiple layers is trivial, but the number of terms quickly become unwieldy, adding little benefit to the reader’s understanding of our work. As explained in “Experimental Setup”, the networks used contain two hidden layers with a width of 512 each. Evaluations on these networks are use for the plots in all figures. Our additional evaluations on 3D implicit surface regression using a network with 6 fully connected hidden layers with a width of 256 each. In this case, we find that the same eigenvalue spectrum trends hold, with the MPE greater than the FFE which is, in tern, greater than the baseline, unencoded network.

Hello. We wanted to see if you have been able to read through our responses as well as the updated draft. We would be happy to address any other concerns and would appreciate any additional feedback.

Hello. Are there any other concerns or questions that we can address? We believe that the updated draft has improved the presentation and demonstrated that the theory holds well for deeper networks (both 2 layer and 8 layer). If possible, it would be helpful to receive any additional feedback soon, so that there is enough time to respond before the deadline on November 26th. Thank you again for the time and effort taken to review our work.

Thank you for your thoughtful feedback on our submission. We greatly appreciate the time and effort taken to review our work. We have carefully addressed the concerns and questions in your original review, and believe that the latest manuscript is much stronger and better aligned with the ICLR audience's expectations. If you agree that these points have been addressed and there are no further concerns or questions, we kindly ask that you consider raising our paper's rating.

We hope this message finds you well. As the rebuttal deadline approaches, we kindly ask that you share any additional feedback or clarifications on your comments. Your input is invaluable in helping us address concerns effectively and enhance the strength and impact of our work.

Thank you for your review and providing feedback to improve the impact of our work. We apologize for the confusion in our presentation and seek to address your questions below and in a new draft. We would, however, like to highlight comments from reviewers uZqD and pL4m, which emphasize the clarity and structure of our paper. In response to the weaknesses presented, the connection between the eigenvalues and high frequency information has been explored in multiple works (Tancik et al., 2020; Wang et al., 2020) and is explained in section 3. Equation 7 shows that the error decays along each eigenvector based on the magnitude of the corresponding eigenvalue. If the eigenvalue is larger, then the decay is faster. The smaller eigenvalues inherently match to higher frequency information, as explained in Basri et al., 2019. Beyond, the answers below, is there any additional information to be included in the paper to improve the reviewer's understanding of the background?

While we fix these notational issues, are there other concerns that would improve your confidence in our work? Having a stronger theoretical understanding of encodings is important to the machine learning community, as it allows for the creation of improved coordinate based MLPs and wider adoption of the MPE. Our work leveraging the NTK provides such an important analysis.

Questions:

1. Dimensions in Equation 1.

Thank you for catching this omission. The dimension of x is as you have stated as it in the original input to the network. The output dimension is then $2dL$. We have included these values next to equation 1 for clarity.

2. Notation in Equation 3.

The plus symbol is a common notation for concatenation.

3. Equation 7 is hard to understand. What are the dimensions of y?

We apologize for the confusion. y in this context all of the training targets concatenated, so it is not just the output dimension. It is the output dimension multiplied by the number of samples. We've included the dimension explicitly in the "Spectral Bias" section and use a capital letter to denote that it’s the entire dataset.

5. Is there an expectation missing in equation 9?

We thank the reviewer for pointing this out. We have updated equation 9, 12, and 14 to include the expectation.

6. Theorem 1 is unclear.

We have removed the $n$ superscript to align more with the notation description. You are correct in that the two are the same. Lambda is a common notation for the eigenvalue of a matrix, and $\lambda_i^{MPE}$ is simply the ith largest eigenvalue of the NTK for the MPE contributions to the kernel shown in equation 13. We will highlight this connection in a new draft.

Hello. We wanted to see if you have been able to read through our responses as well as the updated draft. We would be happy to address any other concerns and would appreciate any additional feedback.

Hello. Are there any other concerns or questions that we can address? We hope that the updated draft has addressed many of the initial concerns about notation and presentation. If possible, it would be helpful to receive any additional feedback soon, so that there is enough time to respond before the deadline on November 26th. Thank you again for the time and effort taken to review our work.

We hope this message finds you well. As the rebuttal deadline approaches, we kindly ask that you share any additional feedback or clarifications on your comments. Your input is invaluable in helping us address concerns effectively and enhance the strength and impact of our work.