Dear Reviewer 11hA,

Thanks a lot for your constructive feedback. Please find our answers below and let us know if you have follow-up questions.

1. The paper presents minimal novel contributions, and only applies an existing optimizer to PINNs. The novel contribution (randomized Nystrom) is shown to be inferior to the deterministic variant in terms of loss obtained with a given compute budget.

We respectfully disagree with both A) the implication that randomized Nyström is the only novel contribution of our work and B) the implication that the contribution regarding the randomized Nyström method is not significant due to its underperformance.

Regarding point A), applying the Woodbury matrix identity and the SPRING algorithm within the context of ENGD for PINNS are both novel contributions that yield very encouraging results and have the potential to make an impact on the field of PINNs. Although these contributions do not come in the form of entirely new algorithms, they align with the statement in the NeurIPS reviewer guidelines that "originality does not necessarily require introducing an entirely new method. Rather, a work that provides novel insights by evaluating existing methods, or demonstrates improved efficiency, fairness, etc. is also equally valuable."

Regarding point B), the fact that the randomized Nyström approximation does not improve the performance of ENGD is quite surprising and only holds as a result of the algorithmic improvements that we have introduced to the baseline ENGD implementation. In fact, one recent work applied a sketching method to reduce the cost of the original ENGD algorithm and found that relative to that inferior baseline, sketching is highly beneficial [2]. Our work thus overturns previous results by showing that, when ENGD is implemented efficiently using the Woodbury matrix identity, randomization is no longer benefcial.

2. Is there a setting where randomized variants of SPRING are superior to deterministic variants, potentially on a more challenging problem which may require larger batch-sizes?

We attempted but failed to identify any scenarios where randomized methods are superior. As discussed above, this is a surprising finding of our paper which is tightly connected to the algorithmic improvements that we introduced to the deterministic algorithm.

3. The method is only evaluated on extremely simple toy systems (linear PDE with explicit solution)

Echoing our response to Reviewer rnPC, this is a fair point. To address your concern, we evaluated two more PDEs (to be included in the next revision): We consider the 4+1d Heat equation and the 9+1d logarithmic Fokker-Planck equation (a challenging nonlinear equation). For both equations, we use the same batch size and computational budget as in [1] sampling one batch at each step, their neural net with 100k parameters, and tune all algorithms with random search.

The results are as follows:

From these additional experiments we conclude:

1. Both ENGD-W and SPRING achieve state-of-the-art accuracies as in our previous experiments, and the advantage of SPRING is confirmed on the more challenging log-FP equation.
2. In comparison to KFAC [1], we first note that ENGD-W and SPRING now scale to this setting (before, ENGD in parameter space would run out of memory due to the neural net's size). Furthermore, we find that both ENGD-W and SPRING reach 2 to 15-times smaller $L_2$-error than KFAC.

These findings demonstrate that the applicability of our methods is not restricted to toy examples of linear PDEs with explicit solutions. Let us know if this makes sense to you and if you have follow-up questions.

4. The paper claims that SPRING surpasses KFAC performance, but does not contain any data on KFAC. What is the performance of KFAC for these systems?

See the previous response for two new examples where both ENGD and SPRING outperform KFAC. Additionally, for the 100D Poisson equation, KFAC has previously achieved an $L_2$-error of 4.3e-3 with a time budget of 10,000 seconds as shown in [1], while in our work SPRING achieves an $L_2$-error of 3.6e-3 with a smaller budget of only 7,000 seconds and identical hardware.

5. In particular how does [KFAC] fare in the regime of large number of parameters P and large batch size N, where presumable both ENGD (scaling poorly with P) and SPRING (scaling poorly with N) are expensive?

You are right that KFAC scales more favourably in batch size than ENGD-W and SPRING. Our comparison focuses on batch sizes that are used in practise (all settings are from existing papers), and we do not observe this advantage. However, we will make this contrast to KFAC more explicit in the manuscript; thanks for pointing it out.

6. Does the faster convergence and runtime of SPRING unlock new applications of PINNs? Can the authors present results on a non-trivial system (e.g. fluid dynamics, inverse problems) where SPRING allows to improve the SOTA?

We view this as an important question for future works; the current work is focused on developing and demonstrating methods rather than tackling specific applications. Nonetheless, we refer the reviewer to the new heat and log-Fokker-Planck equations, which present new, non-trivial settings.

7. What exactly are the "Naive" optimizers in Fig. 4,5? Is it SVD-based randomized Nystrom? If yes, why does it converge at basically the same rate as your Cholesky-based Nystrom if the latter one is "an order of magnitude faster on GPU"?

This is a very good question and we apologize for the confusion. The figures were intended to show the SVD-based method but actually in their current form the naive methods refer to a fully naive Nyström method which approximates $A$ directly as $(A\Omega)(\Omega^T A \Omega)^{-1}(A\Omega)^T$. This naive implementation is known within the linear algebra community to be unstable, as discussed in [3] and others. In our own (intended to be internal) experiments this method worked surprisingly well (similar to the GPU-efficient stable algorithm), but we do not feel comfortable recommending it due to the known instability, which could become a problem in other settings. Unfortunately, the fix proposed by Frangella et al is not GPU-efficient and performs much worse than both of the methods currently pictured in Figures 4 and 5. This lead us to develop our own Nyström approximation that is both stable and GPU-efficient.

Regarding the figures, it was a mistake to include this naive method and we will remove it in the final version. We will instead show the standard (SVD-based) stable algorithm of [3], so that the relative advantage of our proposed GPU-efficient Nyström method becomes clear. We again apologize for the confusion.

8. The authors write on line 189 "Such a sketch-and-solve approach has significant pitfalls [...]". What are these pitfalls? Is it "just" an inaccurate approximation if the target rank is less than the effective rank?

Yes, the specific issue is that with sketch-and-solve, it is impossible to obtain a highly accurate solution without taking the sketch size to be extremely large, which eliminates any computational savings. We will clarify this point in the manuscript.

References

[1] Dangel, F., Müller, J., and Zeinhofer, M. Kronecker-factored approximate curvature for physics-informed neural networks. Conference on Neural Information Processing Systems, 2024.

[2] Mckay, M. B., Kaur, A., Greif, C., and Wetton, B. Near-optimal sketchy natural gradients for physics-informed neural networks. In Forty-second International Conference on Machine Learning, 2025

[3] Frangella, Z., Tropp, J. A., and Udell, M. Randomized Nyström preconditioning. SIAM Journal on Matrix Analysis and Applications, 44(2):718–752, 2023.

Dear Reviewer nNgV,

Thank you for your constructive feedback. Please find our answers below and let us know if you have follow-up questions.

1. The originality of the paper is a bit lacking. ENGD, SPRING, Woodbury's identity, and Nystrom approximation are all established techniques. The main innovation point of the paper seems to be applying these techniques to the specific setting of PINNs.

We respectfully disagree with the implied concern regarding a lack of originality. According to the NeurIPS reviewer guidelines, originality is not limited to introducing entirely new methods. Rather, “a work that provides novel insights by evaluating existing methods, or demonstrates improved efficiency, fairness, etc. is also equally valuable.” Applying Woodbury's identity, SPRING, and the Nystrom approximation to ENGD for the first time aligns with this broader definition.

2. For the GPU-efficient randomized Nystrom, do you have any concrete comparsion against the original version? For example, you could compare wall-clock time and memory usage profiles for different problem sizes, scaling behavior as problem dimensions increase, and other GPU utilization statistics.

To corroborate our results, we generate random matrices of size $N=5000$ and compare the standard Nyström implementation of [1] with our GPU-efficient Nyström approximation on a GPU. We run the benchmarking procedure and present the results in the following table (we use the pattern Standard Nystrom vs. GPU-efficient Nystrom):

As you can see from the results, the GPU-efficient implementation is at least an order of magnitude faster. We will include this evidence, and more ablations, in our manuscript.

3. Does your method works well with larger networks? All the experiments were conducted the MLP only.

Yes! Since our methods are linear in $P$ and architecture-agnostic (unlike KFAC), there is no barrier to extending them to networks of larger sizes or different types. The only reason we use MLPs is to keep the networks the same as in other papers on ENGD to enable fair comparisons with previous works.

4. I feel like the organization of the paper could be improved. The abstract could be a bit more descriptive so that the reader has a clearer expectation. The contribution could be clearly stated there.

We appreciate the feedback and we would like to make the paper as clear as possible. To help us do so, could you elaborate on what is unclear to you in the current write-up, or what changes you would suggest in order to improve the organization of the paper?

References

[1] Frangella, Z., Tropp, J. A., and Udell, M. Randomized Nyström preconditioning. SIAM Journal on Matrix Analysis and Applications, 44(2):718–752, 2023.

Dear Reviewer ZTVE,

Thank you for your constructive feedback. Please find our answers below and let us know if you have follow-up questions.

1. The term $\phi$ in equation (6) just "appears" out of the blue without explanation, leaving the reader to search further afield for clarification. Moreover while I get that (6) is just the usual regularized least-squares expression, what I'm missing is why this is to be preferred over any number of alternative regularization schemes that would, potentially, correspond to different momentum terms?

In equation (6), the variable $\phi$ is just a dummy variable that we optimize over, so it holds no direct meaning. Furthermore, the addition of $\lambda I$ on the left-hand side of (6) is standard within the natural gradient literature and is equivalent to adding an $L_2$ penalty on the right-hand side.

Aside from this, maybe you are referring to the addition of the SPRING momentum term in equation (7)? The intution of equation (7) is to encourage the update direction to be close to the previous one, but it is true that this could be formalized in many ways. We simply use the same choice as the original SPRING paper [1] since this choice has been demonstrated to be effective.

2. I am also a tad confused by (9). It appears from algorithm 2 that the term is already incorporated into the GPU-efficient Nystrom calculation, yet for some reason the right-side of (9) reads $(\text{nys}(J_k J_k^T) + \lambda I)^{-1}$ and not $\text{nys}(J_k J_k^T + \lambda I)^{-1}$. Is this a typo, or have I misread the text?

Thanks for catching this. There is a typo on Algorithm 2, which creates the confusion. In the "Ensure" statement of Algorithm 2, it should read:

• $\hat{A}_{\text{nys}} = B B^T$ on the left, and
• $(\hat{A}_{\text{nys}} + \lambda I)^{-1}v = \dots$ on the right.

We will rectify the error in the revised version.

References

[1] Goldshlager, G., Abrahamsen, N., and Lin, L. A Kaczmarz-inspired approach to accelerate the optimization of neural network wavefunctions. Journal of Computational Physics, 516: 113351, 2024

Dear Reviewer rnPC,

Thanks a lot for your constructive feedback. Please find our answers below and let us know if you have follow-up questions.

1. The experimental evaluation is limited to the Poisson equation, which is relatively simple and may not sufficiently demonstrate the broader effectiveness or advantages of SPRING within the PINN framework. To convincingly establish its utility, the method should be tested on more challenging PDEs, such as the Navier–Stokes equations.

This is a fair point. To address your concern, we evaluated two more PDEs (to be included in the next revision): We consider the 4+1d Heat equation and the 9+1d logarithmic Fokker-Planck equation (a challenging non-linear equation). For both equations, we use the same batch size and computational budget as in [1] and the same neural net with 100k parameters. With these settings we tune ENGD-W and SPRING with random search and compare the results to the results of Adam and KFAC from [1].

The results are as follows:

From these additional experiments we conclude:

1. Both ENGD-W and SPRING achieve state-of-the-art accuracies as in our previous experiments, and the advantage of SPRING is confirmed on the more challenging log-FP equation.
2. In comparison to KFAC [1], we first note that ENGD-W and SPRING now scale to this setting (before, ENGD in parameter space would run out of memory due to the neural net's size). Furthermore, we find that both ENGD-W and SPRING reach 2 to 15-times smaller $L_2$-error than KFAC.

These new results demonstrate that the applicability of our methods is not restricted to the setting of the Poisson equation. Let us know if this makes sense to you and if you have follow-up questions.

2. The paper directly applies the SPRING algorithm, originally designed for VMC problems, to PINN settings without adequately discussing the fundamental differences and similarities between these domains. For instance, VMC methods draw samples according to a distribution proportional to the square of the neural network, making stochastic reconfiguration (SR)-based methods natural and justified. In contrast, PINNs typically use samples drawn from distributions independent of the neural network, raising questions about why SR-based methods remain effective. A deeper theoretical or empirical discussion on this point is needed to clarify the applicability of SPRING to PINNs.

Thank you for raising this point; it is indeed important to clarify why the techniques developed for the VMC community should be applicable to PINNs as well.

For both PINNs and VMC it is well-established that natural gradient methods are particularly well suited [2, 3] and frequently outperform stochastic first-order methods like Adam. This motivates transfer between the fields.

On a more technical level, the probabilistic structure of the typical loss functions in PINNs and VMC are similar: in both cases the loss function is defined by sampling from some distribution $p(R)$, which leads to the ENGD/SR update direction being the solution to a least-squares problem involving that same distribution $p(R)$, and as a result these algorithms and their variants such as SPRING should be implemented by sampling from $p(R)$ as well. In the case of VMC the distribution happens to be wavefunction dependent $p(R) = |\psi(R)|^2$, whereas in PINNs the distribution is a mixture of the uniform distributions over the interior and the boundary of the domain, but this difference does not affect the applicability of SPRING.

3. If it is possible to implement other kinds of second-order optimizer with woodbury identity?

Yes, in fact Quasi-Newton methods like BFGS use the Woodbury identity by default, see for example the Wikipedia page of BFGS (here called Sherman-Morrison formula, a special case of the Woodbury formula). The vanilla Newton method (using the full Hessian) on the other hand cannot be accelarated with the Woodbury formula.

References

[1] Dangel, F., Müller, J., and Zeinhofer, M. Kronecker-factored approximate curvature for physics-informed neural networks. Conference on Neural Information Processing Systems, 2024.

[2] Müller, J., and Zeinhofer, M. Achieving High Accuracy with PINNs via Energy Natural Gradient Descent, International Conference on Machine Learning, 2023

[3] Carleo and Troyer, Solving the Quantum Many-Body Problem with Artificial Neural Networks, Science, 2017