Learning Input Encodings for Kernel-Optimal Implicit Neural Representations

All the Table/Figure Rx can be found in https://anonymous.4open.science/r/ICML-Re-3333

Q1: The proposed method should be applied to other more challenging real-world problems.
A1: Thank you for your valuable suggestion. We have conducted additional experiments on more complex problems, specifically focusing on 3D neural fields and comparing our methods with Instant-NGP, as NeRF can be time-consuming. We performed experiments on the NeRF Synthetic datasets with 25, 50, and 100 view perspective samples. As shown in Table R3, the average PSNR for 25 view perspectives indicates that the PEAK algorithm outperforms the vanilla Instant-NGP. Notably, the improvement is more pronounced with fewer training samples, demonstrating that PEAK effectively enhances generalization capability. Furthermore, Figure R3 shows that PEAK reduces artifacts caused by sparser samples, indicating its ability to leverage the internal structure of the data to improve performance in downstream tasks with limited data. We will include this 3D experimental data in the revised manuscript to further demonstrate the effectiveness and generalizability of our method.

All the Table/Figure Rx can be found in https://anonymous.4open.science/r/ICML-Re-3333

Q1: A comparison of model size and training/inference time, with a fixed number of trainable parameters, is necessary for a fair evaluation.
A1: We have conducted a comparison regarding parameter numbers and training time. Our results demonstrate that PEAK converges quickly and achieves the highest PSNR at the same training time, as seen in Figure R1. Additionally, as shown in Figure R2, PEAK maintains the highest PSNR with an equivalent number of parameters compared to other methods. These findings indicate that our PEAK excels in both efficiency and performance.

Q2: It may be beneficial to relax the language around alignment with the optimal kernel.
A2: Thank you for your insightful feedback. We agree that complete alignment with the optimal kernel is unrealistic due to its unidentifiability from discrete data. Our method aims to endow the INR with properties shared by the optimal kernel, rather than asserting full alignment. In the revised paper, we will adjust our wording to more accurately reflect the actual capabilities of the method and avoid overemphasizing alignment.

Q3: Some notation is not clear.
(1) The notation for the variable A in Theorem 3.3 and Section 3.4 appears inconsistent and requires clarification.
(2) The notation for the embedding function $\gamma$ requires clarification.
(3) The notation for the circled plus is used without definition and can represent multiple operations.
A3: Thank you for pointing out the confusion regarding the notations.
(1) We sincerely apologize for the oversight. In fact, the $A$ in Theorem 3.3 and Section 3.4 represents entirely different concepts. In Theorem 3.3, $A$ denotes a subset of the real numbers, reflecting the range space of a kernel, whereas in Section 3.4, $\mathbf{A}:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R}$ (denoted in bold) is a function that measures the similarity between input points. To enhance clarity, we will change the notation in Theorem 3.3 to signify a set, and we will move the definition of the function $\mathbf{A}$ in Section 3.4 to its first mention in the revised manuscript.
(2) The initial definition of $\gamma$ is provided at line 166 in the right column. We will review the text to ensure clarity and reinforce the understanding that $\gamma$ is a learnable function with specific properties.
(3) The $\oplus$ and $\otimes$ represent the direct sum and direct product, respectively. We will add more detailed explanations following these symbols.

Q4: Encourage the release of the code upon publication, if not before.
A4: We will make our code publicly available upon publication to facilitate reproducibility and further research in this area.

Q5: The supplement on ablation studies and model architecture details should be included in the main paper.
A5: We appreciate your suggestion regarding the supplementary material on ablation studies and model architecture details. We will ensure to reference this material in the main paper and summarize the key findings to provide readers with a clearer understanding of our experimental design.

Q6: Improvements are needed for the clarity of Figure 1.
A6: Thank you for your feedback on Figure 1. We will follow your suggestions to improve its clarity in the revised version. In Figure 1(a), we will reduce the transparency to better distinguish overlapping dots and lines. In Figure 1(b), the "orange arrows" indicate the training progress, i.e. training INR with the vanilla loss function and PEAK, resulting in $f_{\boldsymbol{\theta}}(\gamma(\cdot))$ and $f_{\hat{\boldsymbol{\theta}}}(\hat{\gamma}(\cdot))$, respectively. The "green arrows" represent the application of the NTK theorem to calculate the corresponding NTK values, where $f_{\boldsymbol{\theta}}(\gamma(\cdot))\rightarrow K$ (depicted as the black triangle) and $f_{\hat{\boldsymbol{\theta}}}(\hat{\gamma}(\cdot))\rightarrow \hat{K}$ (depicted as the orange star). The "blue arrows" illustrate the theoretical analysis that introduces the optimal kernel $K^*$ (blue star). The vanilla INR's corresponding $K$ (black triangle) is far from the $K^*$ (blue star). Our KAR aligns the $\hat{K}$ with $K^*$ in the kernel space, guiding the INR to optimize $\gamma$. And the level sets represent the kernel space. We will add further explanation and simplify Figure 1(b) to enhance clarity in the revised version.

Q7: Some typos/minor grammatical mistakes. E.g. the sentence on line 214 (left column) is not complete.
A7: Thank you for pointing out the incomplete sentence. We will conduct a thorough review of the manuscript to correct these typographical and grammatical errors.

All the Table/Figure Rx can be found in https://anonymous.4open.science/r/ICML-Re-3333

Q1: The paper should compare PEAK with SIREN, MFN, BACON, Gauss, WIRE, FINER and SAPE, and explore whether the framework can be applied to and improve these methods.
A1: Thank you for the concern regarding the experimental validity. Actually, we have shown that our PEAK algorithm is not sensitive to the choice of INRs with different activation functions, as illustrated in Appendix Figure 6(c). Furthermore, we have applied PEAK to the INRs you mentioned, except for SAPE, in Tables R1 and R2. The exclusion of SAPE is due to its significantly different training strategy, which requires additional fine-tuning for a fair comparison. For more details about these experiments and our motivations, please refer to A1 to Reviewer QEM1.

Q2: The time and memory requirements of the algorithm are not discussed.
A2: As shown in Figure R1, for the "Baboon" image reconstruction task, our proposed PEAK achieves a PSNR of 30 dB in 2 seconds, which benefits from the kernel alignment providing additional self-similarity to accelerate convergence. As shown in Figure R2, PEAK achieves a higher PSNR compared to other methods while using the same number of parameters. We will include a more comprehensive discussion of these results in the revised version.

Q3: Additional experiments on tasks like 3D shape representation and neural fields.
A3: Thank you for the suggestion. We have conducted more experiments on neural fields, comparing our methods with Instant-NGP, as NeRF can be time-consuming. Specifically, we performed experiments on the NeRF Synthetic datasets with 25, 50, and 100 view perspective samples, respectively. As shown in Table R3, the average PSNR for 25 view perspectives indicates that the PEAK algorithm outperforms the vanilla Instant-NGP. Notably, the improvement is more pronounced with fewer training samples, demonstrating that PEAK effectively enhances generalization capability. Figure R3 shows that PEAK reduces artifacts caused by sparser samples, indicating its ability to leverage the internal structure of the data to improve performance in downstream tasks with limited data. We will include this 3D experimental data in the revised manuscript to further demonstrate the effectiveness and generalizability of our method.

Q4: The supplementary shows some sensitivity to hyperparameter choice. This should be discussed in a limitations section.
A4: Due to space constraints, we only discuss the polynomial degree of $\gamma$ in the main text, as it is unique to PEAK. The influence of other parameters (e.g., the regularization coefficient $\lambda$) is a common consideration for all regularization-based methods. The ablation study on activation functions demonstrates that PEAK is generally insensitive to these choices. While the output dimension $r$ of the regularization network appears to affect the final results, all configurations still outperform the baselines. We will provide the PSNR values of baselines in the supplementary material to avoid potential misunderstandings.

Q5: An analysis of the proposed regularizer's impact on low-frequency bias in INRs is needed.
A5: As shown in Table R4 and Figure R4, we conducted experiments using a synthetic image. We sampled a $256\times 256$ grid from $[-1,1]\times [-1,1]$ as $\mathcal{X}$ and generated $\mathcal{Y}$ using \mathbf{y}_i=\sin(50\pi\sin(\frac{\pi}{3}\cdot\left\\|\mathbf{x}_i\right\\|_2))\in\mathcal{Y}. We then trained MLP, Fourier, and our proposed PEAK algorithm on $\mathcal{X}\times\mathcal{Y}$. This synthetic image represents a frequency gradient transitioning from low (outer) to high (center, $(0,0)$). The results indicate that the MLP struggles to learn higher frequencies even after 10,000 epochs. The Fourier shows some improvement by initially capturing low frequencies before gradually learning high frequencies, but this process is relatively slow. In contrast, our PEAK algorithm effectively addresses the low-frequency bias, enabling it to learn both low and high frequencies almost simultaneously.

Q6: Clarify the number of additional parameters introduced by the second (compact) INR and compare it to a vanilla INR with the same number of parameters.
A6: The second (compact) INR introduces around additional $\frac{1}{10}$ of the first (main) INR parameters. We have compared its performance against a vanilla INR with the same total number of parameters. As illustrated in Figure R2, our algorithm exhibits superior performance.

Q7: "While" should be removed; a reference to Figure 3 is missing.
A7: Thank you for pointing out the incomplete sentence and the missing reference to Figure 3. We will make the necessary corrections in the revised manuscript.

Q1: The choice of the current baselines needs clarification.
A1: We appreciate your inquiry regarding the choice of baselines. Our PEAK algorithm is designed to find an encoder $\gamma(\mathbf{x})$ that enhances the generalization ability of the composed function $f_{\boldsymbol{\theta}}(\gamma(\mathbf{x}))$. Consequently, we focus on these baselines that also work with $\gamma(\mathbf{x})$. The vanilla MLP with $\gamma(\mathbf{x})$ serves as a benchmark for basic performance. The Fourier feature and Hash encoding are among the most influential methods for improving high-frequency representation capability and convergence speed. In response to your feedback, we have extended Table 1 to include a broader range of baselines in Table R1, such as SIREN, MFN, BACON, Gauss, WIRE, and FINER. While these more recent methods show improved high-frequency representation ability in their original studies, their generalization ability remains limited, as shown in Table R1. Additionally, our KAR can be directly applied to these baselines in a plug-and-play manner. We further evaluate the improvement of KAR on the aforementioned baselines in Table R2. The results indicate that KAR enhances the performance of all these baselines, and the numerical results are consistent and not sensitive to the choice of baseline, as KAR estimates the same optimal kernel under the same sampling pattern and image.

Q2: Some theoretical results were not properly referenced.
A2: Thanks for your insightful suggestions from the perspective of a peer in this specialized field. Our main contributions are the introduction of KAR regularization and the PEAK algorithm, both derived from Theorem 3.3. Given the wide-ranging applications of INR across various research domains, our goal is to present the core ideas in a manner that is accessible to researchers with varying levels of prior knowledge, without necessitating consultation of the original literature. Therefore, we have distilled the essential part of these Theorems. In the revised manuscript, we will include proper citations in Appendix A to facilitate readers in finding the related works.

Q3: The per-image results in tables in the main are unnecessary and could be aggregated.
A3: Thank you for pointing this out. We believe that including more image results is important as the performance of INRs is closely tied to the specific characteristics of each image. For instance, in Table 1, PEAK shows the highest improvement on the "Baboon" image with a missing patch, due to its left-right symmetry. This symmetry allows PEAK to enhance generalization by effectively learning internal self-similarities within the signal. A more detailed discussion on this will be provided in the revised version.

Q4: The relationship of the work to the findings of Yuce et al. (2022) should be discussed.
A4: NTK is a useful mathematical tool for analyzing the dynamics of INRs, and many researchers have utilized it to illustrate the properties of INRs. Yuce et al. (2022) analyze INRs from a dictionary learning perspective, which differs significantly from our approach. In contrast, Tancik et al. (2020) provide an earlier analysis of INRs from the NTK perspective, proposing the Fourier feature encoder to improve high-frequency representation capabilities, which we have cited in our paper. To our knowledge, our PEAK is the first work to employ kernel alignment for guiding encoder design. In the revised manuscript, we will include a citation to clarify the similarities and differences between our work and that of Yuce et al. (2022).

Q5: The readability of the paper could be improved.
A5: We recognize that the paper's density may pose a challenge to readers. To enhance readability, we will move technical proofs to the appendix and focus on providing clearer high-level intuitions in the main text, such as summarizing key concepts and illustrating them with examples. Additionally, we will work on simplifying the exposition of our methods and results.

Q6: Notice that, for fully connected networks, depth does not help generalization in the NTK regime (Bietti & Bach, 2020).
A6: Thank you for your valuable comment regarding the findings of Bietti & Bach (2020). We agree that while depth may not theoretically improve generalization in the NTK regime for fully connected networks, in practical applications—especially when the network width is not infinite—depth can still positively influence model performance. In fact, there is very limited research that relies solely on a single-layer INR for realistic applications. Therefore, we propose an encoder learning algorithm rather than modifying the activation function in a single-hidden-layer INR, as done by Simon et al. (2022).