Discussion on training time

---

Thanks for your overall recognition of our method, experimental design, and analysis. Following your suggestion, we provide a further discussion about training time of our method as a supplement to Table 6 in our Appendix. Please refer to our response “Time and memory analysis” to Reviewer 7Wmr. The complete results can be found in: https://anonymous.4open.science/r/iga-inr-icml2025-3A4D/time_memory.md. Overall, IGA averages 1.56× the training time and 1.34× the memory of the baseline but achieves greater improvements (on average, 2.0× those of prior FR and BN, as shown in the main text). Compared to the vanilla adjustment method by full eNTK matrix, IGA saves at least 1/4 of the training time. Thus, IGA improves fitting accuracy by enhancing training dynamics without significantly increasing time.

The additional training time overhead comes from the matrix decomposition. As you mentioned, computing the eNTK and performing eigenvalue decomposition for the population data is intractable due to both memory consumption and computational complexity. Thanks to our theoretical foundations (Theorem 3.1 & 3.2), we only need to decompose the eNTK matrix of the sampled data and inductively generalize them to the population data, which allows for a significant reduction in computational cost while maintaining the effectiveness of spectral bias mitigation. Table 6 in our Appendix shows that a small amount of the sampled data can also lead to a nontrivial improvement.

Discussion on potential solutions

---

Thanks for your constructive advice. We acknowledge INT as a potentially insightful approach and have cited it in the line 014 (right column) of our paper. We will include a following discussion of such methods in the final version. However, the problems that INT aims to address and its use of eNTK are different from ours. INT aims to address the costly training of INRs via the sample selection, while our IGA aims to improve the spectral bias of INRs via the training dynamics adjustment. Based on the nonparametric teaching perspective, the eNTK in INT is used to compute the functional gradient；samples with larger functional gradients are prioritized for selection. Therefore, the specific values are not required for INT, only the relative magnitudes matter. They found that the functional gradient is positively correlated with the discrepancy between the MLP output and the target function; thus, direct computation of the eNTK can be avoided.

While, for IGA, based on the linear dynamics perspective, eigenvalues of the eNTK matrix control the convergence speeds of the MLP on the corresponding eigenvectors; the uneven distribution of eigenvalues leads to varying convergence speeds across feature directions, which is considered as one of the potential causes of spectral bias. IGA aims to uniform these eigenvalues of the underlying eNTK matrix during training to balance the convergence speeds and overcome spectral bias. Therefore, explicit computation of specific eigenvalues and the corresponding eigenvectors on the eNTK matrix is required for IGA.

As you mentioned that the computational cost of the eNTK is enormous, IGA adopts the inductive generalization of the eNTK-based gradient transformation matrix estimated from the sampled data. This avoids the computation of the eNTK matrix for the entire dataset, significantly reducing computational overhead, while still providing a non-trivial improvement. Compared to the vanilla method, i.e, full eNTK matrix, this saves at least 1/4 of the training time, as mentioned in the answer "Discussion on training time".

Response for writing-related comments

---

Thanks for your patient review of our theorem and proofs. $\Theta(\cdot)$ in Theorem 3.2 of the main text denotes a tight bound. Concretely, we take the $\epsilon_2=\Theta(m^{-3/2})$ in line 215, right column as an example. $\epsilon_2=\Theta(m^{-3/2})$ indicates that as the variable $m$ grows large, $\epsilon_2$ behaves asymptotically like $m^{-3/2}$ up to constant factors. That is, as stated in the formal version of Theorem 3.2 (in Section A.4 of the Appendix), $\epsilon_2=\frac{\eta k^3 R_0^2}{m^{3/2}}$, where $\eta, k, R_0$ are constants. Following your suggestions, we will remove the extra parenthesis in Theorem 3.1 (line 186) for readability, define $g_i(\lambda)$ before Equation 2, and refine our explanation of spectral bias for clarity.

Answer for 3D shape and radiance field results

---

Thanks for recognizing our experiments and evaluation approach. We further analyze IGA by comparing it with Fourier Reparameterization (FR) and Batch Normalization (BN), two key training dynamics adjustment methods for mitigating INR spectral bias. For 3D shape task, compared to the vanilla baseline models, FR and BN achieve an average IoU improvement of $9 \times 10^{-4}$ and $6 \times 10^{-4}$, respectively, across five scenes. IGA further achieves an IoU improvement of $4.1 \times 10^{-3}$, which is on average 5$\times$ that of FR and BN. For the neural radiance field, FR and BN achieve 0.12dB and 0.14dB in PSNR, respectively. The average improvement of IGA is up to 0.24dB, which is nearly 2$\times$ that of FR and BN, as mentioned in line 417-418, right column. Please note that the improvements achieved by all these methods are non-trivial, especially considering that they are achieved without altering the network inference structure or inference speed.

Clarification of theorems

---

Thanks for recognizing our theoretical contributions and careful reading. We apologize for any ambiguity in the statement. For Theorem 3.1, there are two assumptions. The first one is that $v_i^{\top}\tilde{v}_i >0$. This assumption is easy to satisfy in practice, as both $\tilde{K}$ and $K$ are real symmetric matrices, and $\tilde{K}$ gradually converges to $K$, leading to increasingly similar eigenvectors. The second one is that $\eta < ( )$. This assumption is made to satisfy the requirements of the linear dynamic analysis. For Theorem 3.2, the statement is indeed meant to convey that max(quantity A in absolute value, quantity B in norm) < epsilon. As you suggested, removing the brackets '[]' and reordering $p$ would make the expression clearer and exactly result in the $j_i$-th row of $\tilde{K}_e$. We will incorporate the above suggestions into the final version.

Discussion about the Lipschitz functions $g_i$

---

Thanks again for careful reading of our theorems and insightful comments. We will revise the Eq. 2 to provide more exposition of $g_i$. Initially, the functions $g_i, i=1,..., N$ are introduced to represent the potential eigenvalue adjustment functions. During the analysis, we found that a Lipschitz continuity condition on these functions serves as a simple yet effective condition for the equivalence between $\tilde{K}$-based and $K$-based adjustments. This condition ensures that, for a set of N Lipschitz functions, there exists a specific Lipschitz constant upper bound, allowing for the existence of a width $m$ such that all adjusted eNTK eigenvalues converge to their corresponding potential adjusted NTK eigenvalues. Following this theoretical analysis, In practice, we simply map the largest $n$ eigenvalues of the $\tilde{K}$ to the $(n+1)$st largest eigenvalue of $\tilde{K}$, keeping the remaining eigenvalues unchanged (please refer to Eq. 5 in our paper). The corresponding operation for the potential corresponding $K$ is mapping the largest $n$ eigenvalues of the $K$ to the $(n+1)$st largest eigenvalue of $K$, and others remaining unchanged. In this setting, the potential errors of these mapped $n$ eigenvalues is dominated by the $(n+1)$st largest eigenvalue. As the eigenvalues from the $(n+1)$-th largest to the smallest are kept unchanged, it follows that their $g_i$functions are the Identity function, with a Lipschitz constant equal to 1. Our wide results and empirical analysis show that the aforementioned simple $g_i$ setting effectively balances the convergence speeds of different frequency components.

Answer for relation to Broader Scientific Literature

---

Thanks for your thoughtful comments. The second paragraph indeed aims to highlight the distinction between training dynamics-based methods and architecture-based approaches such as SIREN and WIRE. As you pointed out, our method can be combined with architectural modifications for further benefits, providing a stronger justification. We will revise this in the final version.

Refinements on Writing

---

Thanks for your valuable writing suggestions. As you pointed out, the first two contributions at the end of the introduction may appear somewhat redundant. We appreciate your suggestion to distinguish theoretical and algorithmic contributions. We will revise the manuscript to better highlight our theoretical insights. Follow your suggestions, we will highlight the empirical NTK approximation earlier and restructure Section 3.1 into a “Background” section. Fig. 9 will be resized to match Fig. 10, and ground truth will be added to Fig. 4. Abbreviation definitions will be included in captions, and line distinctions in Fig. 3 will be adjusted. Minor typos and redundancies such as "improves" will also be carefully revised.

Time and memory analysis

---

Thanks for suggesting a more detailed time and memory requirements analysis. We conducted a measurement of the training time and memory of IGA across all experiments. For example, in the 1D simple function approximation, the vanilla SIREN takes 28ms/iter (ms/iter denotes milliseconds per iteration) with 1009MB memory; SIREN+IGA takes 45ms/iter with 1575MB memory; adjustments (w/o IGA) by the full eNTK matrix, i.e., the eNTK matrix obtained from the population data, takes 198ms/iter with 5881MB memory. In the 2D image approximation, the vanilla PE takes 74ms/iter with 3609MB memory; PE+IGA takes 87ms/iter with 4167MB memory. The full eNTK adjustment requires >100GB memory, which is practically infeasible. In the 3D shape representation, the vanilla ReLU takes 113ms/iter with 2163MB memory; ReLU+IGA takes 163ms/iter with 2231MB memory. In the 5D neural radiance fields, the vanilla NeRF takes 160ms/iter with 9451MB memory; NeRF+IGA takes 286ms/iter with 11033MB memory.

The complete results can be found in: https://anonymous.4open.science/r/iga-inr-icml2025-3A4D/time_memory.md. Overall, IGA averages 1.56× the training time and 1.34× the memory of the baseline but achieves greater improvements (on average, 2.0× those of prior FR and BN, as shown in the main text). Compared to the full eNTK matrix, IGA saves at least 1/4 of the training time and memory. Thus, our method improves fitting accuracy by enhancing training dynamics without significantly increasing time or memory. Notably, since IGA achieves the same results with fewer iterations, it often allows us to achieve results comparable to baseline methods in less or even shorter time. We present examples in the above link.

Practical issue

---

Enhancing network performance by improving training dynamics without affecting the inference holds significant practical value. It helps improve inference performance in cases with limited computational resources. NTK theory has made significant progress in analyzing the training dynamics and spectral bias of INR and shows the theoretical potential to adjust dynamics. However, directly applying it to most INR models for better performance remains impractical due to the intractability of NTK matrix. IGA provides two practical solutions: first, it proves and validates the effectiveness of eNTK (which is generally computable) in adjusting training dynamics; second, it introduces inductive generalization of the transformation matrix derived from the eNTK matrix of sampled data to adjust the overall training dynamics. With these two solutions, IGA achieves greater improvements (on average, 2.0× those of prior FR and BN, as shown in the main text) without significant additional costs—requiring only 45% more training time and memory on average compared to the baseline. Please note that FR and BN also cost more training time and memory but with only minor improvement. Thus, IGA is a practical and effective training dynamics adjustment method for INR tasks.

Additional experiments

---

Thanks for suggesting more models to further show IGA's practical usability. As you mentioned, we have included 2D results of IGA with Gauss, WIRE, FINER in our Appendix and now provide 3D shape results. Due to time constraints and considering that MFN is the backbone of BACON, we extend IGA to MFN for both 2D and 3D tasks. Furthermore, for the neural radiance fields, we extend IGA to the well-known DVGO. The results are listed in the table of the following link: https://anonymous.4open.science/r/iga-inr-icml2025-3A4D/new_exp.md. Previous training dynamics methods, i.e., FR and BN, are also compared. From the numerical results, IGA consistently outperforms FR and BN across 2D, 3D, and 5D tasks, aligning with the main paper. These results further support our claims. We will add these results in the main paper.

Chamfer Distance

---

Thanks for insightful comments about 3D shape representation. Following your suggestion, we computed Chamfer Distance using the code from “Occupancy Networks” and reported the results in the link: https://anonymous.4open.science/r/iga-inr-icml2025-3A4D/chamfer_distance.md. As you mentioned, Chamfer Distance and IoU can behave differently. Despite this difference, IGA still outperforms the baseline and other training dynamics adjustment methods, FR and BN.

Discussion on 3D Shape Representation

---

Thanks for valuable suggestions on the 3D task. The setting of 3D shape representation in the main text is simple but has been adopted by many classic INR works to test the representation capability of INR models, such as PE, SIREN, MFN, Gauss, WIRE, and FINER. Recent works on training dynamics of INR, such as FR and BN, have also used this setting to evaluate the effectiveness of their methods. Therefore, this setting is suitable for demonstrating that IGA can better improve training dynamics to enhance INR’s representation performance for 3D shapes.