Efficient Certification of Physics-Informed Neural Networks

We thank the reviewer for the time spent on the paper as well as for their comments.

(W1) On extending $\partial$-CROWN to certify higher-order PINNs. We partially agree with the reviewer in that extending $\partial$-CROWN to higher-order derivatives following the more efficient, hybrid scheme that we introduced for the first and second derivatives could be laborious as the derivations of higher-order derivatives become more complicated as a result of the chain rule. However, it is theoretically possible to achieve it by following a similar process.

Following the reasoning from Raissi et al., (2019), we can view higher-order derivatives as a depth-wise extension of the original network (as portrayed also in Figure 1 of our paper). As widely observed by the network verification community, deeper networks are more difficult to verify as bound propagation tends to become looser in incomplete verifiers (Gowal et al., 2018; Wang et al., 2018; Shi et al., 2020; Xu et al., 2020a). We mitigate this problem by introducing Greedy Input Branching to obtain tight bounds on the PINNs used in a more efficient way than other verification methods (see Appendix C). For higher-order PINNs it is likely that one will need (i) tighter relaxations of the nonlinearities of the networks, and (ii) more efficient branching methods that allow us to compensate for the tightness loss in deeper networks.

We will modify the last paragraph of Section 5.1 to clarify these points.

(W2) On low residual errors and high solution errors. We thank the authors for the reference to Wang et al. (2022). Regarding the point on Figure 2 the reviewer makes, we would point out that while the residual loss values are of the order of $10^{-2}$, using the causal method the authors of the paper are able to achieve residual loss values below $10^{-5}$ (Figure 4 of the paper), which is what is typically expected in PINN training. This supports the observations we make in Section 6.2. And note also that these values are taken at fixed points instead of taking the maximum across the full domain, which might be significantly higher than at the optimized points.

We partially agree with the reviewer, in that a low maximal residual error does not guarantee a low maximal $\ell_2$ error due to the nonlinearity of the PDEs. We only claim that certified bounds on the boundary, initial conditions and residual error can be interpreted as tolerances in numerical solvers for the PDE – this is explicitly mentioned in Section 6.2.

The reviewer suggests that we “test [our] theoretical framework in some tough cases where PINN encounters propagation failure.” In practice this is what we do in the case of Allen-Cahn’s and the Diffusion-Sorption equations, where the residual errors are so large that we are effectively identifying these PINNs have not been trained properly. We analyze this result in more detail in Appendix C.

We hope these answers are useful to the reviewer in reconsidering the score attributed, and look forward to continuing the discussion.

Additional references:

• Wang, Sifan, Shyam Sankaran, and Paris Perdikaris. "Respecting causality is all you need for training physics-informed neural networks." arXiv preprint arXiv:2203.07404 (2022).

We thank the reviewer for the time spent on the paper as well as for their comments.

(W1) On the relation of residual and solution error. This is an important point. We believe the analysis in Section 6.2 and Appendix C indicates a strong correlation between the two types of errors, but there is more to be done in this direction in future work.

(W2) On Figure 2. The main purpose of Figure 2 is to give an intuition to readers regarding the landscape of the solution and residual error of each of the PDEs studied. However, we agree with the reviewer that the figure would likely be better suited for the appendix, and will move it there in the final version of the paper.

(W3) On running time of MC sampling and $\partial$-CROWN. In general, Monte Carlo sampling is much faster than $\partial$-CROWN for the number of samples we have considered in Table 1 (e.g., inference on $10^6$ samples takes approximately $0.21s$ for Burgers’ equation), scaling linearly with the number of samples for higher values. However, as discussed in the paper, MC sampling only provides a lower bound for the value of the error, whereas $\partial$-CROWN provides a certified upper bound. We will add the runtime information to Table 1.

(W4) On Figure 1. We thank the reviewer for the feedback and will consider a few ways of making the figure more compact.

(W5) Including N_b ablation in the main text. Given the plan to move Figure 2 to the Appendix as per (W2) of this rebuttal, we will move the ablation on N_b to the main text in the final version of the manuscript.

(W6) Typo in Algorithm 1. We thank the reviewer for pointing out that typo which will be corrected in the final version of the paper.

We thank the reviewer for the time spent on the paper as well as for their comments.

(W1) On the assumptions and applicability of $\partial$-CROWN. Regarding the reviewer’s first point on the representation capacities of feedforward networks, we note that these networks are, in theory, universal approximators in the infinite width limit, so they should be able to represent arbitrary solution functions given an appropriate choice of activation. However, we agree with the reviewer in that it might not be an optimal architecture choice for some PINNs (which is likely the reason behind at least some training failures). Despite this, it is widely used in the previous literature, and hence why we chose to focus on these types of networks.

Note that some previous works providing guarantees for PINNs such as Shin et al., (2020), Wang et al., (2022b), or Mishra and Molinaro (2022) only provide them for specific families of PINNs, whereas our work is more general since it can be applied to all PINNs represented by FNNs.

(W2) On the greedy nature of our method. Our method provides a guaranteed upper bound on the global errors over a domain – so the upper bounds provided in Table 1 are guaranteed bounds. We introduce greedy input branching in Section 5.3. to improve the tightness of the bounds obtained using $\partial$-CROWN in the full PDE domain by splitting it into subdomains. Since the union of all the subdomains corresponds to the full PDE domain, taking the worst-case bound of each of the subdomains gives us a bound on the error in the full domain. As such, the bounds obtained via greedy input branching are guaranteed to be globally satisfied over the domain - there is no "approximation" that breaks the bound. More details on the procedure of greedy input branching is available in Appendix H.

(W3) On further experiments. The PINNs in the experiments section were chosen primarily because they are well established in previous literature in the field, from the classic Burgers’ and Schrödinger’s equation in Raissi et al. (2019a) to more recent and difficult ones to learn in the Allen-Cahn’s equation from Monaco and Apiletti (2023) and the Diffusion-Sorption from Takamoto et al. (2022). The HJB equation from [2] is modeled as an MLP, so our framework could be applied to this problem, but as the reviewer mentions the scale of the network would be challenging for $\partial$-CROWN as it is over $1,500\times$ bigger than the networks considered. This drawback of our method is explicitly mentioned in the Section 7 of the paper, and we hope future work will help scale verification methods to those larger networks.

(Q1) On the efficiency of our method. To the best of our knowledge, $\partial$-CROWN is the first framework designed to bound the errors of general PINNs. However, for completeness of analysis, in Appendix B we extend Interval Bound Propagation (IBP) (Gowal et al., 2018; Mirman et al., 2018) – known for its simplicity and trading-off bound tightness for speed – to this setting. The results are presented in Table 3 of our paper (Appendix B). It presents the performance of IBP and $\partial$-CROWN on the initial, boundary and residual errors for a fixed runtime limit in Burgers’ equation. This is a fair comparison which takes into account the runtime/tightness trade-off of the two methods. We observe that $\partial$-CROWN is significantly more efficient than IBP, achieving bounds that are $165 - 1,566\times$ tighter than the baseline in the same total runtime.

(Q2) On takeaways for Theorem 1 and 2. While the expressions for the linear lower and upper bounds are quite efficient to compute in practice, we found it difficult to include them in the main text without excessive notation. However, we agree that the dependencies of the linear functions are not explicit in the theorem, and that a proof sketch should be added in the following paragraph. We will make these modifications in the final version of the paper.

(Q3) On notation. We thank the reviewer for pointing this out, and we will make the suggested adjustment in the final version of the paper.

We hope these answers are useful to the reviewer in reconsidering the score attributed, and look forward to continuing the discussion.

We thank the reviewer for the time spent on the paper as well as for their comments.

(W1) On the motivation of PINN certification and efficiency. We stress in the introduction that the main goal of this paper is to provide a formal guarantee for every input within the feasible domain of the PINN. While most previous works report computed mean and/or max $\ell_2$ solution error computed over a finite number of points, this cannot be relied on in practice as they might not sample domain points where the error is large. We showcase this in the Diffusion-Sorption example in Table 1, where the residual bound estimate using $10^4$ MC samples is $1.1\times 10^{-3}$, while the certified upper bound is obtained at $21.34$. Through sampling we are only able to obtain lower bound estimates on the error which, as we can see from Table 1, can be quite distant from the actual error at specific domain points (visible on the same equation when we increase the number of samples from $10^4$ to $10^6$). This highlights the need for certified upper bounds, which we provide with $\partial$-CROWN.

As far as we are aware, our framework is the first one that can be used to certify arbitrary PINNs with solutions modeled by fully connected neural networks. And while runtime is still a drawback of our framework, in Appendix B we perform an efficiency comparison with an adaptation of Interval Bound Propagation (IBP) (Gowal et al., 2018; Mirman et al., 2018) – known for its simplicity and trading-off bound tightness for speed – to this setting. We compare the performance of IBP and $\partial$-CROWN on the initial, boundary and residual errors for a fixed runtime limit in Burgers’ equation. This is a fair comparison which takes into account the runtime/tightness trade-off of the two methods. We observe that $\partial$-CROWN is significantly more efficient than IBP, achieving bounds that are $165 - 1,566\times$ tighter than the baseline in the same total runtime.

(W2) On the linear bounds and higher-order derivatives. As we note at the beginning of page 6 of the manuscript, $\partial$-CROWN can be extended to higher-order derivatives in a similar fashion to the process used for deriving the second-order bounds. As such, it is not a special case of second-order PINNs, and it can be applied to higher-order ones too. The linear bounds are obtained by relaxing the nonlinearities using linear overapproximations, so they can be applied to any function defined over a continuous domain.

(W3) On theorem notation. The bounds obtained in Theorem 1 and 2 can be computed from the weights of the solution network $u_\theta$ (that is $W^{(k)}$) and the bounds provided by Assumption 1 ($\mathbf{A}^{(k),L}$, $\mathbf{a}^{(k),L}$, $\mathbf{A}^{(k),U}$, $\mathbf{a}^{(k),U}$, $y^{(k), L}$ and $y^{(k), U}$), using the recursive definition provided in Appendix E. While these expressions are quite efficient to compute in practice, we found it difficult to include them in the main text without excessive notation. However, we will change Theorem 1 and Theorem 2 to clarify the dependencies of the linear bounds.

(W4) On the choice of PINNs. The PINNs in the experiments section were chosen primarily because they are well established in previous literature in the field, from the classic Burgers’ and Schrödinger’s equation in Raissi et al. (2019a) to the more recent and difficult ones to learn in the Allen-Cahn’s equation from Monaco and Apiletti (2023) and the Diffusion-Sorption from Takamoto et al. (2022). Another important factor in the choice was the availability of either the code or trained models from all of these previous works. While we considered other suitable higher dimensional PINNs, such as several of the Navier-Stokes equations from Jin et al., (2021), or the Gray-Scott system from Giampaolo et al., (2022), neither training code nor the pre-trained models were released that allow us to apply $\partial$-CROWN.

We hope these answers are clear and useful to the reviewer in reconsidering the score attributed, and look forward to continuing the discussion.

Additional references:

• Giampaolo, Fabio, et al. "Physics-informed neural networks approach for 1D and 2D Gray-Scott systems." Advanced Modeling and Simulation in Engineering Sciences 9.1 (2022): 1-17.
• Jin, Xiaowei, et al. "NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations." Journal of Computational Physics 426 (2021): 109951.