How does PDE order affect the convergence of PINNs?

In order to provide a concise response to theoretical questions, we structured our proof:

• (S1) Determine the condition for the positive definiteness(PD) of the initial Gram matrix.
• (S2) Establish an upper bound for the initial loss in terms of a polynomial of the initial weights.
• (S3) Identify a radius of an open ball within which the Gram matrix preserves PD.
• (S4) Compute the GF and demonstrate that the flow converges within the ball.

Q1. The most critical concern is ...

• I strongly recommend ...
  R: We agree with the concern that the experiments in Section 5 do not fully validate our theorems as they use Adam.
  In response to the feedback, we conducted experiments using the GD. As achieving convergence with GD for higher-order PDEs is challenging, we compare for the heat equation in Eq. (18). Furthermore, in accordance with the request of Reviewer M6m4, experiments were conducted on the convection-diffusion equation. We train models via GD with a learning rate(lr) of 1e-1. PINNs on the convection-diffusion equation employ a reduced lr of 1e-2 as it diverges with a lr of 1e-1. Figures A3 and A4 of the attachment present fast and remarkably stable convergence of VS-PINNs with smaller variance.
  Moreover, we have designed experiments with GD that can clearly support our theory. A detailed description and discussion can be found in the common response.
  We believe these provide further validation of our findings. The results of the experiments in Section 5 may suggest that results to those with GD can be achieved with Adam although we have not proven.
• A limit of Adam ...
  R: We append experiments using GD.
• extend to non-GF-based ...
  R: We coud apply our theory to SGD-like optimization. In the context of stochastic approximation theory (SAT), errors on a gradient originate from the randomness is interpreted as noise when estimating the GF. Following the SAT, the noisy GF converges to a fixed point of the clean GF. Consequently, our theory could be extended.
  However, we acknowledge that S1 is not applicable when collocation points are resampled. In S1, we use the fact that gradients at each collocation point are linearly independent if width is sufficiently large, which does not hold as there are infinitely many collocation points in distribution. Assuming S1, we expect that the S2-S4 are applicable, given that expectation forms and uniformly bounded losses are employed.
• As mentioned above, ...
  R: We acknowledge that GF and GD exhibit different dynamics. However, we believe our theory can be adapted to GD, given backward analysis, including [Miyagawa, 2022].
  Theorem 5.1 and its corollaries in [Miyagawa] show that GF and GD dynamics differ in the limit $t\rightarrow\infty$, assuming a scale-invariant layer is present. Our theory, however, is based on MLPs without normalization layers, and thus does not meet the condition in the theorem.
  Instead, we could apply Theorem 3.3 from [Miyagawa], which states that the GD dynamics follows GF with a counter term. Since the term is also a polynomial, we anticipate that S2-S4 can be adjusted. In S1, we expect our induction on a maximum degree can be adapted because the leading term occurs only in the counter term. Therefore, we expect to prove the convergence of GD based on the backward analysis.

Q2. I would recommend that ...
R: We apologize and thank you for pointing this out. We will rectify it by including the 'utils' module and providing a requirements.txt file to specify the necessary dependencies. We regret any inconvenience this may have caused.
Q3. ReLU^p is called the RePU
R: We appreciate your recommendation to cite the references related to the RePU. We will include these citations.
Q4. Generalize to other activation?
R: S1 effectively exploits the non-smooth nature of the RePU. This approach can be expanded to include other non-smooth activation functions like Leaky RePU. Additionally, it seems feasible to bound the loss function using a polynomial of the parameters. Therefore, extending the S2-S4 to this type of activation function seems plausible.
However, our approach in S1 is not applicable for smooth activations and it is difficult to establish the PD of the Gram matrix. This leads some studies to assume the PD of the Gram matrix and analyze the convergence of the optimization flow. Assuming S1, our results would extend to activation functions whose derivatives are polynomially bounded.
Q5. Generalize beyond two-layer?
R: S2-S4 of our theory would remain feasible as the composition of polynomials is a polynomial. However, the non-smooth point can be entangled, making the applicability of S1 less straightforward.
Q6. Is there any other evidence that ...
R: There are theoretical papers that analyze the impact of activation in terms of optimization convergence. For instance, [Panigrahi, 2020] computes the eigenvalue of the Gram matrix for both non-smooth and smooth activations. The authors provided evidence indicating that when data span a low dimensional space, the smallest eigenvalue of the Gram matrix can be small leading to slow training. It is also demonstrated that the smallest eigenvalue of the Gram matrix associated with non-smooth activations is larger than that of smooth activations under minimal assumptions on data. Consequently, the findings imply that training will be more rapid for non-smooth activation.
Q7. Seem to be sufficient condition
R: Although they are sufficient conditions, the leading terms originate from the PD of the Gram matrices, which is a prerequisite for convergence. Given that the PD of the Gram matrices inherently depends on the PDE and p, we contend that the insights regarding the detrimental impact of large p could be derived from the inequalities.
References Panigrahi et al., Effect of Activation Functions on the Training of Overparametrized Neural Nets, ICLR, 2020.

Thank you for your time and effort. The authors’ response addressed my concerns, particularly regarding the validity of the experiments, and answered my questions clearly. I encourage the authors to incorporate all the discussion from their response into the manuscript. I have increased my score from 3 to 5.

Q1. There is no sharpness of the lower bound $m>C$. Thus, ...

R: We acknowledge the reviewer’s concern regarding the sharpness of the bound we presented in our paper. We concede that we have not demonstrated the sharpness of the bound in our theorems. However, it is important to note that the leading term of the bound we derived is based on conditions necessary for the Gram matrix to be positive definite, which is a crucial property of the Gram matrix for ensuring the convergence of the gradient flow (GF) to a global optimizer. Since the Gram matrix is defined by the PDE loss and the network structure, factors such as the PDE order of the power of the ReLU activation inherently affect the positive definiteness of the Gram matrix.

Therefore, even though the bound we provide may not be sharp, we believe it can still offer valuable insights into how reducing the order and power improves convergence. It underscores how variations in PDE order and ReLU power affect convergence, thus providing the first theoretical demonstration of the relationship between the PDE order and the convergence of PINN, which has been empirically observed frequently. We believe this is an important step in understanding the relationship between PDE order, ReLU power, and the convergence of PINNs.

However, there is a limitation in that we have not proven the sharpness of the bound, and as you point out, we agree that this should be clearly stated in the manuscript. We will revise the manuscript to acknowledge this limitation and adjust our argument accordingly.

Q2. Achieving zero training loss is one step in ...

-Q2-1. [1] The article discusses error analysis/coercivity estimates for PINNs..

R: We would like to express our gratitude for your comprehensive and insightful feedback on our manuscript. We are particularly appreciative of the list of references and the valuable insights you have provided, which help to contextualize our work within the broader framework of mathematical analysis for PINNs.

In order to successfully employ the PINN approach for solving PDEs, it is essential to ensure that the following four conditions are met:

• (C1) The network is capable of approximating the solution to the PDE.
• (C2) The minimizer of the PINN population loss is the solution to the PDE.
• (C3) The minimizer of the empirical loss approximates the minimizer of the population loss.
• (C4) The minimizer of the empirical loss is obtainable. The universal approximation theory of the network addresses C1, while the existence of exact solution to the PDE on compact domain underpin C2. Consequently, theoretical analyses of PINNs are primarily centered on the generalization error analysis of C3 and the optimization error analysis of C4.

The papers you suggested focus on C3, which is concerned with examining the conditions for the convergence of the generalization error. In contrast, the present paper targets C4, namely the demonstration of the conditions for convergence of empirical loss. Besides, our result is based on the impact of the order of the governing PDE, whereas the suggested paper employed the coercivity of the PDE.

Despite this, there is a paucity of theoretical research in this area. Therefore, it is imperative to analyze C3 and C4 based on the characteristics of the given PDE.

As you noted, [1] examines generalization error by capitalizing on the coercivity of the PDE operator. On the other hand, Our work delves into the optimization error analysis in conjunction with the order of the PDE. While theoretical studies on PINN optimization exist, few have investigated the impact of the inherent nature of the PDE being solved. A substantial body of empirical evidence indicates that training PINNs becomes increasingly challenging as the PDE order increases, underscoring the significance of our research as a foundational step in this area.

Nevertheless, we concur with your point that a considerable number of steps remain in order to achieve a comprehensive understanding of the fundamental principles underlying PINNs. We believe that extending ideas in the vein of [1] could facilitate an investigation into the influence of the differential order of the PDE operator on the generalization error, thereby advancing our understanding.

We would like to reiterate our gratitude for your invaluable feedback and for sharing these references. We are eager to incorporate your suggestions to enhance our manuscript.

-Q2-2. The benefit of first-order reformulations of PINN type losses is ...

R: As the reviewer mentioned, both [1] and [3] examine the advantages of splitting PDEs into first-order formulations. In contrast to Reference [1], which examines the benefits from the perspective of S3, our paper focuses on how variable splitting enhances optimization in S4 by reducing the differential order included in the loss function. Extending our theoretical framework to analyze the impact of the variable splitting strategy on the generalization error of PINNs, as suggested by [1], represents an intriguing and significant research direction. We believe that incorporating discussions on these references would enrich our paper. Therefore, we plan to include them and the additional discussions in the revised version of our manuscript. We are confident that these enhancements will contribute to the depth and breadth of our work.

Thank you again for your invaluable feedback and suggestions. We appreciate your guidance in improving our manuscript.

Q1. Practical efficiencies are expected ...

R: Table A1 in the attachment shows GPU memory, running time (the mean of the 50 epochs), and the number of model parameters that correspond to experiments presented in our paper. Becasue VS-PINNs need as many networks as auxiliary variables, finer VS-PINN requires more parameters to be trained. However, its reduction of the differentiation order in the loss significantly reduces memory consumption, despite the additional loss terms. Regarding running time, splitting does not increase the overall training cost.

Q2. All the conclusions in Section 3 can be ...

R: We want to clarify that the conclusions of Section 3 of our paper are not directly derived from the previous work [22], as the bound in [22] is not dominated by the PDE and the power of ReLU$^p$. On the other hand, our paper attains tighter bounds than [22]. Moreover, the dominating term of the proposed bounds is determined by PDE order and activation power, allowing us to observe the impact on GF convergence. To provide a more comprehensive understanding, we will delineate the leading terms of the bound of [22] and ours in the following paragraphs. We hope that the reviewer will carefully consider why our tighter bound is insightful.

• (Leading term in [22]) In [22], Theorem 3.8 states that if the width $m$ is $\tilde{\Omega}\left(\delta^{-3}\right)$ then GF converges to a global optimizer of PINNs for second-order linear PDEs. The leading term of the bound is $\delta^{-3}$, which stems from the product of triple $\delta^{-1}$. One $\delta^{-1}$ came from the Markov inequality to bound initial loss at Section B.4. The remaining two $\delta^{-1}$ are required for the bound of $R_w$ at Eq. (74) and $R_a$ at Eq. (76), which are also derived from the Markov inequality immediately following Eq. (73). Therefore, the leading term $\delta^{-3}$ originated from the Markov inequality and is independent of the PDE order or the activation power.
• (Leading term in ours) In contrast to using the Markov inequality, we obtain the bounds in a deterministic way; refer to Proposition C.6 and Lemma C.5 for the initial loss and $R_w$, respectively. This yields a more precise bound for our Theorem 3.2, whose dominant term is contingent upon the power of the activation function and, thereby on the PDE order. Consequently, we obtain a considerably tighter bound that depends on the PDE order, whereas the previous bound from [22] was not affected by the order. From the relationship between the PDE order and the bound for convergence, we could derive a theoretical insight that reducing the PDE order enhances the convergence behavior of PINNs.

Q3. How about performance in typical PDEs?

R: In response to the reviewer's request, we conducted experiments on a convection-type equation. Since observing the effect of VS on first-order PDEs is unable, we convection-diffusion equation $u_t+u_x-u_{xx}/4=0$, whose exact solution is $u\left(t,x\right) = e^{-\frac{1}{4}t}\sin\left(x-t\right)$. We trained PINNs with $p=3$ and VS-PINNs with $p=2$, using same settings with the heat Eq. (18) in the manuscript. Figures A2 and A3 in the attachment show that VS-PINNs reach lower train loss and achieve more stable convergence for both GD and Adam.

Q4. The equation in line 247 has a typo..

R: The symbol actually denotes an integer set defined on line 531 of the Appendix, which did not clearly specify in the main manuscript. We apologize for any confusion this may have caused. We will make sure that the necessary definition is included in the revised version of the paper.

Q5. How to choose $L$ in practice?

R: We agree that discussing the optimal choice of $L$ is crucial for effectively utilizing VS-PINNs. However, identifying the optimal $L$ is challenging because training deep learning models involves a multitude of complex considerations.

Our paper is focused specifically on one aspect of deep learning training: convergence of GF. According to our theory, splitting a given PDE into a system of first-order PDEs, such that $| \xi | =1$, helps GF most likely to converge. This finest splitting results in a loss that includes only first-order derivatives, allowing the use of $p=2$, which in turn improves the convergence of GF the most.

Experiments also show that breaking down a PDE into a system of lower-order PDEs tends to be most feasible for convergence. Moreover, Table A1 which was referenced in the previous response indicates that VS is an effective approach for optimizing GPU memory utilization. However, while finer splitting enhances convergence, it also increases the number of functions that need to be parameterized by networks. Therefore, in practice, the choice of L should be balanced based on the governing PDE and memory/computational constraints.

Furthermore, it is critical to consider other factors, so we acknowledge that this is an area requiring further exploration for a robust analysis of the optimal $L$.

Q6. There are some more powerful and advanced backbones, ...?

R: We conducted experiments to assess the efficacy of our approach within the suggested PINNsFormer architectural framework. We implemented our VS strategy, conducting experiments based on the wave equation in Section 4 of the original paper of PINNsFormer [1]. All configurations specified in [1] were adhered to, including the use of the L-BFGS optimizer. To further examine the behavior of models, we also experimented with alternative optimizers, namely Adam and SGD, with a learning rate of 1e-4.

The results are depicted in Figure A5 of the attachment. As evidenced by the results, the VS does not yield a notable impact on the PINNsFormer framework for all three optimizers. It seems that the Transformer architecture is substantially different from the MLP structure examined in our paper, resulting in disparate outcomes and indicating that comparable results may not be attainable in this context.

Q1. The numerical results pertaining to the parameter ...

R: The theoretical results presented in this paper indicate that networks with a lower power $p$ have a higher probability of convergence. A reduction in $p$ would facilitate optimization, which may be observed as an acceleration in the convergence process. It should be noted, however, that our findings pertain to an improvement in the probability of convergence rather than to the speed of convergence.

We acknowledge that the experiments presented in our paper could be misinterpreted as demonstrating faster convergence. To address this, we conducted additional experiments to provide a more robust validation of the theoretical results. Details of these experiments can be found in the common response above. The experimental results indicate that as p increases, a wider network is required for convergence, which is consistent with our theoretical findings. These results will be included in the revised manuscript, and Section 5 of the submitted manuscript will be revised to clarify the existing experiment and avoid misinterpretation of the results as showing the effect of acceleration.

Q2. The derived bound demonstrates the substantial impact of ...

R: We are grateful for the reviewer's thoughtful review and for emphasizing the necessity for empirical validation regarding the relationship between network width $m$ required for convergence and power $p$ of activation. We devised an experiment to validate the impact of power $p$ and PDE order $k$ on the width size $m$ required for convergence, which can be found in the common response above. Please see the description and our discussion in the common response above.