Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks

Thank you for taking the time to review our article and for your insightful comments. We apologize for the oversight in notations. We agree that adding some context before diving into derivations would improve readability, especially for a broader audience. In the revised version, we will make sure to clarify the definitions of the notations when they are first introduced and add more introductory context.

In this work, for gradient descent, we improve both the requirements for the learning rate $\eta$ and the width $m$. Furthermore, we establish convergence guarantees for natural gradient descent (NGD) in training over-parameterized two-layer PINNs, demonstrating that NGD achieves: (1) an $\mathcal{O}(1)$ learning rate, and (2) faster convergence compared to gradient descent.

Although this is a purely theoretical paper, we can provide some experiments to justfy our theoretical results. When we are able to revise our paper, we will add the detailed experimental results as a separate section in the manuscript, and the code to reproduce the experiments will be added to the Github.

We conduct experiments on three problems, the 2D Poisson equation with reference solution $u _{ref}=\sin(\pi x)\sin(\pi y)$, the 1D Heat equation with reference solution $u _{ref}=e^{-\frac{\pi^2t}{4}}\sin(\pi x)$ and the 2D Helmholtz equation with wave number $k=4$, and the reference solution $u _{ref}=\sin(\pi x)\sin(k\pi y)$. All codes are conducted by the Pytorch framework. The configurations used in these examples are listed in Table 1. We report the relative $L^2$-error of the NGD optimizer, the SGD optimizer, the Adam optimizer, and the L-BFGS optimizer in Table 2. The relative $L^2$-error is defined as follows: $\frac{||\hat{u}-u _{ref}|| _2}{||u _{ref}|| _2},$ where $\hat{u}$ denotes the predicted solution and $u _{ref}$ represents the reference solution. $N _f$ is the number of interior sampling points, and $N _b$ is the number of boundary sampling points.

Table 1: Configurations of Different Equations

Table 2: Relative $L^2$-error of Different Optimizers

In all the experiments，we run the NGD and L-BFGS method for 500 epochs, while the SGD and Adam are trained for 10, 000 epochs. The loss decay during training demonstrating that the NGD method converges significantly faster than other optimization methods.

Table 3 presents the convergence performance of the NGD method with different learning rates on the 2D Poisson equation. The experimental results demonstrate that NGD maintains stable convergence across a wide range of learning rates without significant degradation in final accuracy.

Table 3: Relative $L^2$-error Comparison Across Different Learning Rates for NGD method

In addition, a comparative analysis of the model performance is performed with progressively increasing network widths. Table 4 presents the variation of $L^2$-error with respect to network width for 1D Poisson equation with $u_{ref}=\sin(4\pi x)$. The results demonstrate that increasing network width leads to accuracy improvements.

Table 4: Relative $L^2$-error Comparison Across Different Network Width for NGD method

From an experimental perspective, NGD demonstrates rapid convergence during the training process. Compared to other optimization algorithms, it requires significantly fewer epochs to converge. Furthermore, the experimental results illustrate the strong robustness of the NGD method with respect to hyperparameter selection. Therefore, the empirical findings validate our theoretical conclusions.

We sincerely appreciate the reviewer's time and valuable feedback on our manuscript. Let us address your questions point by point.

Q1: Relations to other methods.

A1: We apologize for the insufficient background discussion of Natural Gradient Descent (NGD) in our work. Although NGD shares some similarities with the classical Gauss-Newton method, key differences exist. For instance, in our problem, the Gauss-Newton iteration is given by $w(k+1)=w(k)-(J(k)^T J(k))^{-1} J(k)^T u(k),$ whereas the NGD iteration follows $w(k+1)=w(k)-\eta J(k)^T (J(k) J(k)^T )^{-1} u(k).$

Müller and Zeinhofer proposed ENGD for PINNs and empirically demonstrated its ability to yield highly accurate solutions. However, NGD formulation differs from ENGD, and they did not provide theoretical convergence guarantees.

Q2: Lack of simulations.

A2: Thanks for the suggestion. We provide some experiments to validate our theoretical results. In the revised manuscript, we will add a separate section presenting detailed experimental results, and the code to reproduce the experiments will be added to the Github.

The configurations used in these examples are listed in Table 1. We report the relative $L^2$-error of NGD, SGD, Adam and L-BFGS optimizers in Table 2. $N _f$ is the number of interior sampling points, and $N _b$ is the number of boundary sampling points.

Table 1: Configurations of Different Equations

Table 2: Relative $L^2$-error of Different Optimizers

In all the experiments，we run the NGD and L-BFGS method for 500 epochs, while the SGD and Adam are trained for 10, 000 epochs. The loss decay during training demonstrating that the NGD method converges significantly faster than other optimization methods.

Table 3 presents the convergence performance of the NGD method with different learning rates on the 2D Poisson equation. The experimental results demonstrate that NGD maintains stable convergence across a wide range of learning rates without significant degradation in final accuracy.

Table 3: Relative $L^2$-error Comparison Across Different Learning Rates for NGD method

In addition, a comparative analysis of the model performance is performed with progressively increasing network widths. Table 4 presents the variation of $L^2$-error with respect to network width for 1D Poisson equation with $u_{ref}=\sin(4\pi x)$. The results demonstrate that increasing network width leads to accuracy improvements.

Table 4: Relative $L^2$-error Comparison Across Different Network Width for NGD method

Q3: Dependence on $d$.

A3: The dimension-related terms in Gao's results are implicitly hidden in their outcomes. However, our results outperform theirs. For instance, when estimating the initial value $L(0)$, for one of the terms $||w|| _2^2\sigma(w^Tx)$, Gao employed the bounded differences inequality, specifically applying $| ||w|| _2^2\sigma(w^Tx)|\lesssim ||w|| _2^3\sim d^{3/2}$. This leads to their estimate of $L(0)$ being at least $L(0)=\Omega(d^3)$. In contrast, our analysis, by leveraging Lemma C.2, achieves a better bound of $L(0)=\mathcal{O}(d^2)$. In their analysis, similar bounding techniques were employed in other parts as well, thus our results concerning the dimension $d$ are superior to theirs.

Q4: Dependence on $f$.

A4: From the analysis, we can see that the initial value $L(0)$ depends linearly on $f$, and the requirement for the width $m$ is $m=\Omega(L(0))$. Therefore, a complicated $f$ leads to a higher requirement on $m$, making convergence more challenging.

We sincerely appreciate the reviewer's time and valuable feedback on our manuscript. Let us address your questions point by point.

Q1: Experimental investigations on NGD for PINNs.

A1: Thanks for the suggestion. We provide some experiments to validate our theoretical results. In the revised manuscript, we will add a separate section presenting detailed experimental results, and the code to reproduce the experiments will be added to the Github.

The configurations used in these examples are listed in Table 1. We report the relative $L^2$-error of NGD, SGD, Adam and L-BFGS optimizers in Table 2. $N _f$ is the number of interior sampling points, and $N _b$ is the number of boundary sampling points.

Table 1: Configurations of Different Equations

Table 2: Relative $L^2$-error of Different Optimizers

In all the experiments，we run the NGD and L-BFGS method for 500 epochs, while the SGD and Adam are trained for 10, 000 epochs. The loss decay during training demonstrating that the NGD method converges significantly faster than other optimization methods.

Table 3 presents the convergence performance of the NGD method with different learning rates on the 2D Poisson equation. The experimental results demonstrate that NGD maintains stable convergence across a wide range of learning rates without significant degradation in final accuracy.

Table 3: Relative $L^2$-error Comparison Across Different Learning Rates for NGD method

In addition, a comparative analysis of the model performance is performed with progressively increasing network widths. Table 4 presents the variation of $L^2$-error with respect to network width for 1D Poisson equation with $u_{ref}=\sin(4\pi x)$. The results demonstrate that increasing network width leads to accuracy improvements .

Table 4: Relative $L^2$-error Comparison Across Different Network Width for NGD method

Q2: The influence of width for convergence results.

A2: Theoretically, the algorithm achieves guaranteed convergence once the network width exceeds a sufficient threshold, with no further improvement in convergence rate from additional width increases. This fact is consistent with established results on gradient descent for PINNs and regression problems. Empirically, wider networks exhibit lower training and test errors.

Q3: Why consider NGD?

A3: From a theoretical perspective, existing convergence analyses of optimization algorithms for PINNs have primarily focused on gradient descent. However, as shown in this paper's improvements on gradient descent, gradient descent imposes relatively stringent requirements on certain hyperparameters (e.g., learning rate) and exhibits slow convergence rates. Therefore, we turned our attention to NGD, which we think may account for curvature information of the function. Theoretically, NGD has more relaxed requirements for the learning rate and achieves faster convergence rates. Experiments also demonstrate that NGD converges more rapidly during training.

As for why L-BFGS was not considered, it is because for non-convex optimization problems like PINNs, L-BFGS is more complex and harder to analyze compared to NGD. Given that recent studies have shown quasi-Newton methods (e.g., SOAP [1]) perform well in optimizing PINNs, analyzing the convergence of such algorithms for PINNs will be an important future direction. Additionally, more efficient implementations or variants of NGD also represent a key area for future research.

[1]: Vyas N, Morwani D, Zhao R, et al. Soap: Improving and stabilizing shampoo using adam[J]. arXiv preprint arXiv:2409.11321, 2024.

We sincerely thank the reviewer for their valuable time and constructive suggestions on our work.

Q1: Methods to help the PINNs' design.

A1: Thanks for the kind suggestion. The results of the paper motivate us to establish "good" Gram matrix $H^{\infty}$ to reduce the strictly learning rate requirement for GD optimizers, and this can be achieved by properly design the loss function, the network architecture, and so on. For example, since the learning rate requirement for GD is related to $\mathcal{O}(1/||H^{\infty}||_2)$, existing works [1] have adaptively adjusted the weight in the different loss components of PINNs, to improve the eigenvalue distribution of the Gram matrix $H^{\infty}$, thus we can use a normal learning rate to accelerate PINNs' training and convergence.

[1]Wang S, Yu X, Perdikaris P. When and why PINNs fail to train: A neural tangent kernel perspective[J]. Journal of Computational Physics, 2022, 449: 110768.

Q2: Add experiments to verify theoretical findings.

A2: Thanks for the suggestion. Here, we provide some experiments to validate our theoretical results. In the revised version, we will add the detailed experimental results as a separate section, and the code to reproduce the experiments will be added to the Github.

We conduct experiments on three problems, the 2D Poisson equation with reference solution $u_{ref}=\sin(\pi x)\sin(\pi y)$, the 1D Heat equation with reference solution $u_{ref}=e^{-\frac{\pi^2 t}{4}}\sin(\pi x)$, and the 2D Helmholtz equation with wave number $k=4$, and the reference solution $u_{ref}=\sin(\pi x)\sin(k\pi y)$. All codes are conducted by the Pytorch framework. The configurations used in these examples are listed in Table 1. We report the relative $L^2$-error of the NGD optimizer, the SGD optimizer, the Adam optimizer, and the L-BFGS optimizer in Table 2. The relative $L^2$-error is defined as follows: $\frac{||\hat{u}-u _{ref}|| _2}{||u _{ref}|| _2},$ where $\hat{u}$ denotes the predicted solution and $u _{ref}$ represents the reference solution. $N _f$ is the number of interior sampling points and $N _b$ is the number of boundary sampling points.

Table 1: Configurations of Different Equations

Table 2: Relative $L^2$-error of Different Optimizers

In all the experiments，we run the NGD and L-BFGS method for 500 epochs, while the SGD and Adam are trained for 10, 000 epochs. The loss decay during training demonstrating that the NGD method converges significantly faster than other optimization methods.

Table 3 presents the convergence performance of the NGD method with different learning rates on the 2D Poisson equation. The experimental results demonstrate that NGD maintains stable convergence across a wide range of learning rates without significant degradation in final accuracy.

Table 3: Relative $L^2$-error Comparison Across Different Learning Rates for NGD method

In addition, a comparative analysis of the model performance is performed with progressively increasing network widths. Table 4 presents the variation of $L^2$-error with respect to network width for 1D Poisson equation with $u_{ref}=\sin(4\pi x)$. The results demonstrate that increasing network width leads to accuracy improvements.

Table 4: Relative $L^2$-error Comparison Across Different Network Width for NGD method

From an experimental perspective, NGD demonstrates rapid convergence during the training process. Compared to other optimization algorithms, it requires significantly fewer epochs to converge. Furthermore, the experimental results illustrate the strong robustness of the NGD method with respect to hyperparameter selection. Therefore, the empirical findings validate our theoretical conclusions.