Fast Training of Sinusoidal Neural Fields via Scaling Initialization

Dear Reviewer jBfM,

We thank you for your concise and constructive suggestions, as well as the effort you put into reviewing our work. Below, we provide responses to the weaknesses and questions you mentioned.

W1. About the novelty issue.

Novelty: My main issue with the paper and what I feel is its main weakness is the fact that it is entirely focused on one type of neural field, namely sinusoidal neural fields. The paper does not say anything about other neural fields in general. The authors do say that their methods would not work on ReLU based neural fields due to ReLU having the positive homogeneity condition and while the authors do say that sinusoidal neural fields are used within the community I would argue that there are several papers showing that sinusoidal neural fields are not as effective as other neural fields, such as [1], [2], [3], [4]. While the paper is written well I do feel that it being entirely focused on one type of neural field means it won't be as useful to many researchers in the neural fields community.

We respectfully disagree with the reviewer’s point that focusing on sinusoidal neural fields significantly undermines the usefulness and the impact of our work. The reason is threefold.

• When initialized with WS, SNF outperforms [1,2,3,4]. It is true that [1,2,3,4] demonstrates that neural fields with other activations outperform SIREN-initialized SNFs. However, our experimental results suggest that the proposed weight-scaled initialization makes the SNFs great again, making it outperform the GaussNet [1], WIRE [2], NFSL [3], and Sinc [4] baselines quite consistently over various setups (Table 1); we have compared with [4] in our response below. Thus, we argue that the claim ‘sinusoidal neural fields are not as effective as other neural fields’ is no longer true.
• SNFs are widely used in practice. We find that SNFs are still widely used in various domains, e.g., [5,6,7,8,9]. In fact, the authors of [1,2,3,4] still utilize sinusoidal activations in their recent work [10]. Thus, we believe that developing an efficient training scheme for SNFs can make a significant practical impact.
• Our work reveals the intimate relationship between the initialization and training cost. The initialization scale of neural nets have been actively studied in both practical and theoretical domains. However, the connection between the training efficiency and initialization is still poorly understood. We strongly believe that our case study on sine-activated networks, which demonstrates that better initialization can greatly improve the training efficiency for free, can inspire much future effort in this area.

Comparison with Sinc. We have additionally compared with sinc-activated neural fields [4] on the Kodak image regression task identical to our setup for Table 1; see the table below. Although the official code was not available online, we were able to contact the authors and retrieve the codes (we are very grateful for the help).

We observe that weight-scaled SNFs remain preferable over the baselines.

We will also include the comparison against sinc on other tasks as well, as soon as we finish running the codes.

W2. Large-scale experiments

Large Scale Experiments: The paper compares its methodology of weight scaling on a tasks such as image regression, binary occupancy, spherical data, audio data and fit a nef (in the appendix). Yet I noticed there were no experiments on Neural Radiance Fields (NeRFs) which I found very surprising. The authors say that their method is very effective at reducing the training time of neural fields which is generally large yet many of the tasks such as image regression, audio, occupancy are not considered big tasks within the community. On the other hand NeRF is a big experiment that takes anywhere between 4 hours to 12 hours to train, depending on the fidelity the user wants. I would have liked to see how their method did on such a task that requires hours of training. I also looked at the papers the authors compare to namely [1], [2], [3] and noticed that these papers compared their methods on NeRF. In the appendix the authors do compare their method on "fit a nef" on a large scale dataset but these takes less than 1 hour to train even with a SIREN. Thus I feel another big weakness of the paper is that the authors did not compare their method on a task that takes hours to train such as NeRF, so that we can see how their method does on a more complicated large scale neural field.

Thank you for the thoughtful suggestion. In response to the reviewer’s comment, we have conducted additional experiments on NeRFs and differential equations, which are more large-scale in terms of training time. Briefly, our method demonstrates better performance compared to the baselines.

Neural radiance fields (NeRFs). We provide the results of the NeRF experiment trained with two different datasets from ‘NeRF-Synthetic’ (as in WIRE). Further details and qualitative results can be found in Appendix F.7.

Solving differential equations. We also performed an experiment on solving wave equations, following the setup of [11]; see Appendix F.8 for the detailed setup. The table below highlights the MSE comparisons (at $t=0$) against the baseline methods. The results (Figure 24) show that weight scaling proves to be effective not only for the given observations but also for previously unseen temporal coordinates.

Q1. The effect of WS on other initializations.

Your paper is entirely focused on neural fields that use a sinusoidal activation. The weight scaling method you propose scales the initialization of a sinusoidal neural field by a certain factor $\alpha$, as per eqn (6) in your paper. This method takes the SIREN initialization as its basis. Could you explain what happens if we take a standard initialization scheme like Kaiming uniform or Xavier uniform and scale it in exactly the same way as your eqn. (6)? Does this lead to anything new? Why did you apply your weight scaling to only the SIREN initialization?

We have applied weight scaling on SIREN initialization, as it guarantees favorable theoretical and empirical properties. SIREN initialization theoretically guarantees that the activation distributions remain the same over the layers [11], and this is true even after the weight scaling, as we formalize in Proposition 1. On the other hand, Xavier / Kaiming initializations are designed for linear / ReLU activations, respectively, and thus does not guarantee the same distribution preserving property for sine-activated neural networks. It has also been empirically observed that such initializations perform poorly on SNFs [11,12].

To make this point concrete, we have evaluated the performance (train PSNR) of weight-scaled Xavier / Kaiming-initialized SNFs on the image regression of a Kodak image (see table below); we train for 150 steps, with all other setups identical to the paper. We observe that Xavier / Kaiming initializations lead to a poor performance, with or without weight scaling.

Q2. The effect of WS on other activation functions.

Could you explain whether your results would go through for other activations that are used within the neural fields literature such as Gaussian [1], Wavelet [2], Sinc [4]?

Thank you for this interesting suggestion.

Theoretically, we do not expect the weight scaling to work satisfactorily with other activations, as we do not have any guarantee that activation distributions will be preserved over the layers. This is in contrast with the sine activations, where WS does not hurt such property, provably (Proposition 1).

Empirically, we have conducted additional experiments and observed that the weight scaling has quite a mixed impact on NFs with other activations. We have tested on a single Kodak image regression task for a single image, as the results are as below.

We observe that weight-scaled GaussNet and Sinc NF also enjoy better training in terms of the training PSNR, but their test PSNR degrades more severely than in SNFs. ReLU+P.E. remains indifferent to WS, and WIRE shows a degraded performance. We have added detailed discussions in the Appendix E.4.

Q3. Changing frequency as $\omega_h / \alpha$.

Your weight scaling essentially just scales the initialization of the inner layers of the initialization by Sitzmann et al for SIRENs. However, if you look at the denominator of eqn. (3), line 194, there is the term $\omega_h$ which is supposed to be the frequency of the sinusoid being used on that layer. Could we not get the same effect as your initialization by using a sinusoid with frequency $\frac{\omega_h}{\alpha}$?

In principle, if we replace $\omega_h$ by $\omega_h/\alpha$ without weight scaling, the initial functional remains the same as SNF initialized with standard SIREN initialization. This is because $\alpha$ terms in the initial weight distribution and the activation function cancel out. In addition, we may expect that the training actually becomes slower; $\omega_h$ is known to serve the role of boosting the weight gradients [11], and thus reducing $\omega_h$ may slow down the training.

To confirm this intuition, we have conducted actual experiments (Appendix E.3). Our empirical results show that such operation makes the training slower than the SIREN-initialized SNF.

Q4. Condition number of Hessian

In appendix C.3 you analyse the condition number of the Hessian of the loss landscape. You mention that the condition number of the Hessian directly affects the convergence rate. However, my understanding is that this is only true for convex loss landscapes. Loss functions associated to neural networks are almost always non-convex, as they employ non-linearities, therefore in such cases the condition number of the Hessian may not be well defined, as the Hessian may have negative eigenvalues or zero as an eigenvalue. Could you comment on this? I don't understand this analysis you have provided in your paper.

Thank you for pointing this out. On the one hand, the reviewer is correct that the theoretical relationship between the convergence rate and the condition number holds strictly for convex loss landscapes. Thus, the expression that “directly affects” holds for the convex case, but not neural nets; we have fixed this expression.

On the other hand, it is also true that there exists a rich body of empirical and theoretical works that ideas from the convex case, especially regarding the Hessian eigenspectrum, also holds true (or at least insightful) in the optimization of deep neural nets. Precisely, [13] shows that the emergence of outlier eigenvalues when training neural nets; [14] shows that the negative eigenvalues disappear quite quickly during the early phase of training, and outlier eigenvalues slow optimization process in neural network; [15,16] shows that the maximum eigenvalue of Hessian (often referred to as “sharpness”) is indeed closely related to the optimal learning rate, similarly to the convex case; [17,18] study the neural net training techniques (batchnorm and residual connection) through the Hessian, which is quite effective empirically. In this sense, Hessian-based analysis provides a useful perspective toward neural net training. We have clarified this point and the distinction from the “provable results” in the convex case in the revised manuscript.

Furthermore, in Figure 11 of Appendix C.4 (previously C.3), we empirically confirm the observations of [14] on SNFs that the negative eigenvalues of the Hessian quickly disappear after a small number of training steps. Within this context, we have measured the ratio of “absolute” eigenvalues as an approximation of the condition number. While this quantity is not strictly equal to the real condition number, the approximation may help provide useful intuitions to the SNF training.

[1] Ramasinghe et al., Beyond Periodicity: Towards a unifying framework for activations in coordinate-MLPs, ECCV 2022

[2] Saragadam et al., WIRE: Wavelet implicit neural representations, CVPR 2023.

[3] Saratchandran et al., From Activation to Initialization: Scaling insights for optimizing neural fields, CVPR 2024.

[4] Saratchandran et al., A Sampling Theory Perspective on Activations for Implicit Neural Representations, ICML 2024

[5] Tack et al., Learning large-scale neural fields via context pruned meta-learning, NeurIPS 2023.

[6] Andreis et al., Set-based Neural Network Encoding Without Weight Tying, NeurIPS 2024

[7] Thrash et al., MCNC: Manifold Constrained Network Compression, arxiv 2024

[8] Ashkenazi et al., Towards Croppable Implicit Neural Representations, NeurIPS 2024

[9] Ma et al., Implicit Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes, NeurIPS 2024

[10] Ji et al., Sine Activated Low-Rank Matrices for Parameter Efficient Learning, arXiv 2024

[11] Sitzmann et al., Implicit neural representations with periodic activation function, NeurIPS 2020

[12] Parascandolo et al., Taming the waves: sine as activation function in deep neural networks, 2016

[13] Sagun et al., Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond, arxiv 2016

[14] Ghorbani et al., An Investigation into Neural Net Optimization via Hessian Eigenvalue Density, ICML 2019

[15] Cohen et al., Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability, ICLR 2021

[16] Zhu et al., Understanding Edge-of-Stability Training Dynamics with a Minimalist Example, ICLR 2023

[17] Lange et al., Batch Normalization Preconditioning for Neural Network Training, JMLR 2022

[18] Li et al., Visualizing the Loss Landscape of Neural Nets, NeurIPS 2018

Dear reviewer jBfM,

Thank you once again for taking the time to review our manuscript. We greatly appreciate your time and effort in helping us improve our work.

As there are only 60 hours remaining in the discussion period, we wanted to ask if you have any remaining concerns or questions that we could address. If so, please do not hesitate to let us know.

Best regards,
Authors.

Dear reviewer jBfM,

Thank you for your continuing efforts to provide useful suggestions to improve our manuscript and also for recognizing our efforts by raising a score.

On not being at the level of ICLR.

We strongly believe that our manuscript makes significant and original contributions that are worth sharing with the audience of ICLR. In particular, the key strengths of our paper are:

1. Contributes new and significant perspective: Our work presents a novel perspective that rethinks the role of initialization for speeding up neural field training. Through our exploration, we challenge the common belief that initial weight scales should be strictly determined to a specific value for distribution-preserving properties, to demonstrate for the first time that weight scales can be tuned for a speedup without losing generalization properties. This finding reveals a new and effective design parameter for a better NF initialization, which may inspire much future studies.

2. Theoretical findings and analysis.: We theoretically show that scaling initialization of SNF does not harm the distribution preserving properties. Moreover, by taking a closer look at weight-scaled functional, we mathematically characterize how the weight scale heavily affects the network properties (i.e., amplifying both relative power of high-order harmonics and early layer gradients) and establish connections to the training speed (i.e., analysis on eNTK).

3. Practical impact: Practically, the proposed weight scaling provides substantial speedup upon the baselines. For instance, on Kodak dataset, training speed is more than 2.9x faster than the best baseline (when measuring #steps until achieving training PSNR 50dB; Fig 25a). As this speedup does not come at the expense of generalization, we strongly believe that our findings bear a significant practical importance.

For these reasons, we sincerely expect that many of the ICLR audience will find our work meaningful and useful.

On [10].

We sincerely apologize for any misunderstanding caused by our response. Our purpose in citing [10] as “in [10], they still utilize sinusoidal activations” was not to emphasize that their proposed network takes the exact form of an SNF, but rather to highlight that their idea shares a similar spirit with that of sinusoidal neural fields: using sine functions as an effective mean to enhance the expressivity.

To elaborate further, the main idea in [10] is to increase the rank of the weights by embedding them within a sinusoidal function with fixed frequency gains. Notably, [10] includes experiments based on neural fields (e.g., NeRF and occupancy field), where sinusoidal functions are applied to the MLP weights, resulting in better performance than a vanilla MLP. This can be interpreted as an attempt to reconstruct the finest details (as well as improving overall reconstruction quality via expanding expressivity) with low-rank matrices, which aligns closely with the practical significance of the sine activation function in SNF.

Therefore, we still believe that sinusoidal neural fields (or at least their core concepts) are widely used in practice, either directly (as mentioned in our previous response) or rather indirectly (as [10]).

Yet another reason to prefer sine activations.

In addition, we note that sine activations can be more preferable for fast training for computational reasons; training with sine activations runs faster on conventional GPUs. This is because their gradients take a simpler form (another sinusoid) than other sophisticated activations, which can be computed fast based on look-up tables (e.g., Volder’s algorithm). To show this point, we have conducted additional experiments to measure the wall-clock time of training a model for 150 steps, on a single NVIDIA A6000 GPU (see below table; averaged over 5 trials). We observe that the sine activations are clearly faster in this aspect. Thus, we still believe that a study on SNF may have meaningful implications, in terms of faster training speed.

We sincerely appreciate your active participation during the discussion phase. If there are any remaining questions regarding our response, please let us know.

Best,
Authors.

Dear Reviewer bvG5,

We thank you for your thoughtful comments and for recognizing the key strengths of our work. Below, we address the concerns you mentioned, point by point.

Q1. Impact on long-term generalization

Impact on Long-Term Generalization : How does WS affect long-term generalization on larger datasets or in cases where SNFs are used for predictive tasks (e.g., continuous function approximation or spatial data interpolation)?

We believe that the reviewer refers to tasks like NeRF (i.e., rendering novel views with interpolated or untrained spatial coordinates) or solving PDEs (i.e., approximating the value of the wave function at unseen temporal coordinates). We have conducted additional experiments on these tasks, which shows that WS indeed improves the performance on these tasks.

NeRF. We have evaluated on two data from ‘NeRF-Synthetic’ (as in [1]). Further details and qualitative results have been added to the revised manuscript, in Appendix F.7.

PDE. We have followed the setting of [2] to evaluate the reconstruction quality of the wave equation with partial observations; the table below presents the MSE comparison at $t=0$. To demonstrate the generalization to further time steps, we have also added the quantitative results in the Figure 24 of the revised manuscript, which also confirms the effectiveness of our method.

Q2. Additional experiments.

It seems that the main difference between this work and the work by Sitzmann et al. (2020b) is the weight scaling, specifically the addition of the factor $\alpha$, correct? In the work by Sitzmann et al. (2020b), several results are presented, but this work has only compared a few of them.

The reviewer is correct that the main difference between our method and the work by Sitzmann et al. (2020b) is how we initialize the SNFs.

Following this suggestion, we have added two experiments that have been missing from our paper and appeared in Sitzmann et al. (2020b): NeRF and PDEs. We have provided experimental results in our response to your previous question, and to the Appendices F.7 and F.8 in the revised manuscript.

Furthermore, we would like to highlight that we also provide two tasks that Sitzmann et al., (2020b) did not cover: Climate data (Table 1), and neural datasets (Appendix F.6). We sincerely believe that this makes our method quite well-validated.

Q3. Another work on scaling homogeneous neural networks

There are some works such as a work by Taheri et al. (2020) which have worked on the scaled neural networks with nonnegative homogeneous activation functions (In particular, ReLU activations).

We thank the reviewer for the pointer. We have included the reference in the revised version.

Q4. About the structure of NN.

In practice, drawing conclusions about a specific structure is challenging for two main reasons. First, conclusions cannot rely on only a few datasets. Second, modifying other aspects of the structure can lead to substantial differences in results. Specifically, changes to a network’s structure—such as the number of hidden layers, the number of hidden nodes, or the learning rate—can significantly impact the outcomes.

In fact, this is exactly why we have conducted a variety of ablation studies, ranging over many tasks and modalities (Table 1), resolution (Fig 6(a)), network width (Fig 6(b)), network depths (Fig 6(c)), and the choice of individual datum (Fig 7), to find that scaling up consistently improves the training efficiency of SNF. In the revised version, we have added new ablations on the choice of the learning rates (Appendix E.5) and additional tasks (Appendix F.7 and F.8).

We agree that a careful validation is needed to draw conclusions on a specific structure, and have put our best effort to do so.

[1] Saragadam et al., WIRE: Wavelet Implicit Neural Representations, CVPR 2023

[2] Sitzmann et al., Implicit neural representations with periodic activation function, NeurIPS 2020

I thank the authors for their efforts. The presentation of the work is now very good, and I have updated the $**presentation**$ rating to "Excellent".

Dear Reviewer svcN,

We greatly value detailed feedback and thoughtful questions, which have helped clarify and enhance the understanding of our work. We provide detailed responses to the questions below.

W1. Theoretical contributions of our work.

The theoretical results are quite limited. For instance, to show that the proposed weight scaling increases the relative power of the higher-frequency bases, the paper considers the effects of such scaling on a 3-layer SNF only for the case when the width is one.

We consider a simplified setup because such simplifications lead us to theoretical claims with clear and concise insight. In fact, it is quite common to introduce dramatic simplifications–-as analyzing a neural network with more than two layers remains quite challenging–such as:

• Infinite width: As in [1,2,3]
• Two-layer nets: As in [4,5]
• Linear activations: As in [6,7]
• Infinitesimal learning rates: As in [8,9]

Through such simplification, our lemma 2 & 3 provides a clear yet fresh perspective on how weight scaling affects the initial spectrum of the SNFs. Furthermore, we would like to highlight that our theoretical results are quite well-aligned with the empirical phenomenon, which is rarely true for many theoretical works with complicated settings.

W2. Discussion about eNTK.

While the optimization trajectories are studied via the lens of the empirical neural tangent kernel, it would be more convincing if there are theoretical/empirical results that quantify the impact of such scaling on the gradient loss and the speed of convergence of gradient descent.

We deeply appreciate this suggestion. Following the comment, we have added additional theoretical analysis on the quantitative nature of the eNTK condition number; see Appendix C.3 of the revised manuscript. In this new appendix section, we analyze the eNTK of two-layer SNF to show that the weight scaling reduces the decay rate of the eNTK eigenvalues by characterizing the scale-dependency of the eNTK eigenvalues. This is well-aligned with the empirical observations we provided that the condition number of eNTK is small (Figure 5 and Appendix C.2). The condition number of eNTK can work as a proxy for convergence speed of the gradient-based training in the kernel regime, as described in [10,11].

Q1. About Figure 2.(b)

In Figure 2(b), the relationships between test PSNR and acceleration plotted for both cases of frequency tuning and weight scaling do not seem to be one-to-one. Why is that so? Am I missing something there?

The reviewer is correct that there is no one-to-one correspondence between individual data points for two plots in Figure 2(b). Essentially, this is because our focus here is to illustrate how tuning $\omega$ and $\alpha$ allows us to trade the training speed for generalization; we have chosen the ranges of $\omega$ and $\alpha$ that can help us best capture the full tradeoff curve, which turns out to be not exactly one-to-one. Indeed, we needed to search for a much wider range for frequency tuning ($\omega \in [10,6510]$) than for weight scaling ($\alpha \in [0.4,4.0]$). We have clarified this point further in the revised manuscript, and added further details in Appendix F.1.

Q2. The effect of Gaussian initialization.

In Eq. (3), the initialization scheme involves scaling the uniform distributions. Will similar weight scaling trick achieve similar acceleration results when the distributions involved are non-uniform (say Gaussian)?

Thank you for the intriguing suggestion. The short answer is: yes.

We first note that, unfortunately, the “Gaussian” version of the principled SIREN initialization does not exist, due to the difficulty in theoretically proving the distribution preserving property theoretically with Gaussian distributions. However, we can conjure up a naïve Gaussian version by replacing the uniform distributions with the Gaussian distribution with the same variance.

We have experimentally tested this variant on a single Kodak image regression task (see the table below). By scaling up the Gaussian-initialized weights, we empirically observe that WS has a similar effect of increasing the train PSNR, and increasing-then-decreasing the test PSNR.

Q3. Training with SGD optimizer.

How does the speed-up offered by the proposed weight scaling scheme depend on/interact with the choice of optimizers (say SGD vs Adam)? Is it more effective for certain choices of optimizers?

We first clarify that we have used Adam because it trains faster in general, and thus fits our goal of fast training.

With SGD, we confirm that weight scaling remains quite effective. The table below presents the image regression results on Kodak, when trained with SGD (the same setting as in Table 1). The weight scaling clearly outperforms the baselines with SGD, but achieves slightly lower performance than with Adam. We note that WIRE exhibits low performance with the SGD optimizer, which may be addressed through further extensive hyperparameter tuning.

Q4. About Figure 3.

For the results in Figure 3, the learning rates vary with layers and are scaled down to match the rates of unscaled SNF. Can you provide more details on how they vary with depth? Also, what if the same learning rate is used for all layers?

For the $k$th layer of an $l$-layer SNF, we have used the learning rate $\eta \cdot \alpha^{l-k}$. This is inspired by our analytic derivations at the end of Section 4.1 (and Appendix B.3). This scaling down makes the effective learning rate of each layer to be similar, whose performance we visualize in Figure 3(b). If we use the same nominal learning rate (i.e., $\eta$) for all layers, then this case is identical to the proposed weight scaling method (Figure 3(a)). We have made this point clearer in Appendix F.2. of the revised manuscript.

Q5. Details on Eq.(10) and trainable $\alpha$.

Can you provide more details on how the optimization problem (10) is solved? Can we treat $\alpha$ as a trainable parameter instead?

In this work, we have explored solving the optimization Eq.(10) by a simple grid search. That is, we have trained weight-scaled SNF with many different values of $\alpha$ and selected the value which maximizes the training speed and meets the constraint. We have tried all $\alpha$ within the range $[1.0,4.0]$ with the grid size 0.2. We have added the details in the Appendix F.3. of the revised manuscript.

Considering trainable alpha. Thank you for an interesting suggestion. We believe that using trainable $\alpha$ to solve Eq.(10) will be quite challenging for two reasons. (1) Even if $\alpha$ is trainable, the function $\mathrm{speed}(\cdot)$ is not differentiable, forcing us to use RL-driven algorithms, as in NAS. (2) Empirically, we observe that training SNF with differentiable $\alpha$ is quite unstable and leads to a poor performance; we have added Appendix E.2 with explicit experiments.

[1] Jacot et al., Neural Tangent Kernel: Convergence and Generalization in Neural Networks , NeurIPS 2018

[2] Lee et al., Deep neural networks as gaussian processes, ICLR 2018

[3] Lee et al., Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent, NeurIPS 2019

[4] Atanasov et al., Neural Networks as Kernel Learners: The Silent Alignment Effect, ICLR 2022

[5] Kunin et al., Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning, NeurIPS 2024

[6] Arora et al., A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks, ICLR 2019.

[7] Min et al., On the Convergence of Gradient Flow on Multi-layer Linear Models, ICML 2023

[8] Chizat et al., On Lazy Training in Differentiable Programming, NeurIPS 2019

[9] Chizat et al., Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss, COLT 2020

[10] Xiao et al., Disentangling trainability and generalization in deep neural network, ICML 2020

[11] Liu & Hui. Relu soothes the ntk condition number and accelerates optimization for wide neural networks. arXiv, 2023

[12] Sitzmann et al., Implicit Neural Representations with Periodic Activation Functions, NeurIPS 2020

Dear reviewer svCN,

Thank you once again for taking the time to review our manuscript. We greatly appreciate your time and effort in helping us improve our work.

As there are only 60 hours remaining in the discussion period, we wanted to ask if you have any remaining concerns or questions that we could address. If so, please do not hesitate to let us know.

Best regards,
Authors.

W4. About fixed iterations.

Fixed Iterations: The choice of number of training iterations to report performance is arbitrary and could reflect unfairly on the baselines. The authors should consider including the PSNR curves for training until convergence (for both WS and baselines) for understanding practical feasibility, since ground truth data to stop at a target PSNR is not available in practice.

Thank you for this valuable suggestion. We have added the full PSNR curve for image regression (for Kodak dataset) to Figure 1 of the revised manuscript, which shows that the proposed method indeed provides a substantial speedup. We are also preparing the full PSNR curve for other tasks, which we are planning to add to the appendix of the revised version.

In addition, we would like to clarify the rationale behind “stopping at fixed iterations.” This criterion is meaningful, as many of the considered tasks are about fitting the seen coordinates (e.g., image regression) where we care about training PSNR. For such tasks, the training PSNR tends to increase slowly but steadily as the training progresses, without a clear convergence point. Thus, for these cases, we find it reasonable to compare the performance after a fixed number of steps. Nevertheless, we agree to the fact that comparing the full PSNR will provide a more informative comparison.

W5. The effect of mini-batch training.

Full Batch Training/Susceptibility to SGD Noise: In practice, it is often not possible to perform full batch training due to size of dataset, network size etc. How does the noise in moving to stochastic (batched) gradient descent affect the convergence of weight scaling? Experiments that reflect those settings will help understand better the limitations of the proposed framework.

TL;DR. Our method continues to outperform the baselines with smaller batches.

We first clarify that we have primarily compared full-batch training quite intentionally, as our main focus is the training speed. However, we agree that in many practical cases where we cannot use full batch; indeed, some of our results are already using mini-batches (e.g., occupancy fields).

To provide further insights, we have conducted additional experiments on mini-batch training. The table below summarizes the results across various batch sizes for the image regression task. For smaller batches, we confirm our method continues to outperform the baselines. We observe that weight scaling consistently performs effectively in the mini-batch setting, further demonstrating the robustness of our method. Detailed descriptions, along with PSNR curves, are provided in Appendix E.1.

W6. Missing citation.

Related work: Missing citations for Multiplicative Filter Networks (Fathony et al, 2021) as well as spectral bias literature.

Thank you for the pointer. We have added the references and discussions to the revised manuscript.

[1] Saragadam et al., WIRE: Wavelet Implicit Neural Representations, CVPR 2023

[2] Sitzmann et al., Implicit neural representations with periodic activations functions, NeurIPS 2020

Dear Reviewer 3nXe,

We sincerely appreciate your valuable feedback and insightful questions, which have enriched our experiments and baselines. Below, we address the questions about our work in detail.

W1. Additional experiments.

The current choice of experiments is rather limited. Since the contribution is largely methodological, experiments on important practical applications of SNFs, specifically NeRF and solving differential equations (for instance, see Saratchandran et al., 2024), as well as larger datasets, are needed to justify the claims.

Thank you for the suggestion. Following this comment, we have conducted additional experiments on NeRFs and partial differential equations (PDEs). The results confirm that our method shows improved performance over the baselines on these tasks as well.

NeRF. We have evaluated on two data from ‘NeRF-Synthetic’ (as in [1]). Further details and qualitative results have been added to the revised manuscript, in Appendix F.7.

PDE. We have followed the setting of [1] to evaluate the reconstruction quality of the wave equation with partial observations; the table below presents the MSE comparison at $t=0$. To demonstrate the generalization to further time steps, we have also added the quantitative results in the Figure 24 of the revised manuscript, which also confirms the effectiveness of our method.

Please let us know if there is another dataset which we should additionally evaluate on–we would be more than happy to prepare them, as long as our GPU resources can handle them within the limited time window.

W2. The reason for choosing Xavier initialization as a baseline

Baseline: The included baseline with Xavier initialization is largely irrelevant from a practical standpoint.

We have included the Xavier init. baseline to highlight the importance of preserving the activation distributions across layers. As [2] points out, classical initializations (e.g., Xavier) fail to retain such properties on SNFs and have poor performance. On the other hand, our initialization preserves the activation distributions, as our Proposition 1 shows. We have clarified this point further in the revised manuscript (Appendix F.4).

W3. Using MFN as a baseline

Baseline: Alternatively, Multiplicative Filter Networks (Fathony et al, 2021) is an important baseline to consider, especially since multiplicative networks, akin to weight scaling, are known to speed up learning for higher frequencies [4] and thus, directly competes with weight scaling in terms of spectral bias.

We highly appreciate this suggestion. Following this comment, we have additionally compared with MFN (see the revised Figure 1 and Table 1). We find that while MFN achieves impressive performance, our proposed approach still achieves better results. In particular, the revised Table 1 is now as follows:

We have also added comparisons with the extended number of iterations, for the image regression task; find these at Table 4 in the revised manuscript.