Learning a Neural Solver for Parametric PDEs to Enhance Physics-Informed Methods

W7 PINO

We apologize, but there seems to be a significant misunderstanding here. We regret if this was not made clear enough in the paper and will clarify the distinction between the objectives and components of the two frameworks. Let us attempt to summarize this distinction below. We hope this will help clear up the confusion.

The PINO paper proposes a method to use neural operators by leveraging both physics-based and data-driven losses. At inference time, the model is fine-tuned using only the physical loss (with an optimizer, such as Adam). In this approach, data are used when available to learn the solution operator, complementing the information provided by the PDE loss.

Our objective is fundamentally different. In our setting, the neural network (NN) is not used to train an ansatz, as in PINO. Instead, it is designed to optimize the convergence of the gradient-based optimizer itself. Specifically, the NN is employed at test time to accelerate the optimization procedure used to solve the PDE. The role of the data is to train this optimizer, guiding the solver rather than the ansatz operator.

Once the solver optimizer is trained, it can be applied directly to new instances of the parametric equation without requiring additional retraining or fine-tuning, as is necessary in PINO (referred to as test-time optimization in the PINO paper). In other words, our work addresses a “learning to solve” problem. This distinction is explicitly stated in lines 80–82 of our manuscript: "With respect to unsupervised approaches, our model implements an improved optimization algorithm, tailored to the parametric PDE problem at hand instead of using a hand-defined method such as stochastic gradient descent (SGD) or Adam."

W8 Trajectory

Trajectory is a spatiotemporal sequence in the observation space of the phenomenon, corresponding to a simulation of a PDE.

Contribution

Again, we apologize if we were not clear enough. We hope that we have highlighted the significant differences between our framework and objectives and those of PINO above, and that this clarification resolves the misunderstanding. Please let us know if you have any remaining questions, which we will be happy to address.

References

[1] Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators.

W1 Unsupervised

There might be some misunderstanding on this point. What is meant by unsupervised in the context of neural PDE solvers is that no labeled data is required (except for some I/B conditions), i.e. the neural network is optimized through the PDE loss only, which is the essence of PINNs. This terminology is commonly used by many authors. We provide below references including some key authors on this topic together with an indication on where to find this wording in the manuscript.

• Sebastian Schaffer, Thomas Schrefl, Harald Oezelt, Alexander Kovacs, Leoni Breth, Norbert J. Mauser, Dieter Suess, Lukas Exl, Physics-informed machine learning and stray field computation with application to micromagnetic energy minimization, 2023: "PINNs are inherently mesh-free and unsupervised learning models." in the abstract.
• Wuyang Chen, Jialin Song, Pu Ren, Shashank Subramanian, Dmitriy Morozov, Michael Mahoney, Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning, NeurIPS 2024 "Unsupervised Pretraining: no simulated solution" (Figure 1) ... "our unlabeled PDE data include physical parameters(aij ,bi,c), forcing functions (f), and coordinates (grids of the discrete physical space).", Section 3.1
• Yuyao Chen, Lu Lu, George Em Karniadakis, and Luca Dal Negro, Physics-informed neural networks for inverse problems in nano-optics and metamaterials "Moreover, PINNs do not require any data on the inverse parameters it predicts, and thus it belongs to unsupervised learning." (section 2)
• Cuomo, S., Di Cola, V.S., Giampaolo, F. et al. Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next. J Sci Comput 92, 88 (2022). https://doi.org/10.1007/s10915-022-01939-z "The basic concept behind PINN training is that it can be thought of as an unsupervised strategy that does not require labelled data" (section 1)
• Edgar Torres, Mathias Niepert, Survey: Adaptive Physics-informed Neural Networks, 2024, "Physics-informed neural networks (PINNs) present a promising solution for solving partial differential equations (PDEs) using neural networks, especially indata-scarce scenarios due to their unsupervised learning abilities", (abstract).
• Klapa Antonion, Xiao Wang, Maziar Raissi, Laurn Joshie,, Machine Learning Through Physics–Informed Neural Networks: Progress and Challenges, "The training process of PINNs functions asan unsupervised strategy, eschewing the need for labeled data stemming from prior simulations or experiments." (page 47)

W2 Neural Operator

• The distinction between operators and mappings between finite-dimensional spaces is not relevant in our context. Here, the FNO is used to learn a mapping between finite-dimensional spaces and not for its operator propperties. Said otherwise, any other network architecture could be employed as well, whether inspired by a neural operator or not (see ablation in Appendix E1, Table 12 of the updated manuscript). FNO was chosen for its robustness across different PDEs, as demonstrated in the experiments.
• Thank you for pointing this out; we have added the definition of the FNO abbreviation (see the updated version of the paper, line 420).

W3 varrho

This was already addressed in the paper in section 3.2 on line 234 "we first transform the gradient using a neural network $F_{\varrho}$ with parameters $\varrho$".

W4 Linear span

The proposed method leverages a linear combination of non linear functions. Its approximation properties are similar to the ones of NNs (the same goes for Fourier basis or different polynomial bases for example used for many surrogate models for PDEs). Note that the neural operator "DeepONet" ([1]) makes use of a similar linear expansion of basis function. This is a choice of implementation and not a simplification. Besides, as detailed in footnote 4 of the paper, our preliminary experiments have shown that using a more complex combination of the basis functions, e.g. through a neural network did not improve the performance and simply made the training more complex. This last point is illustrated in an ablation in Appendix E1, table 11, where we compare a linear and a non linear combination of the basis functions.

W5 Solve

Following your suggestion, we replaced "solve" in algorithm 2 by "Estimate" in order to avoid any confusion. Please see the updated version of the manuscript.

W6 Solve and predict

We agree, F itself is not used as a solver, instead it is used to update the gradient direction within the solver. However, this shorthand greatly simplifies the description. We will make this clear in the final version (footnote 4 of the updated version). Thanks for the suggestion.

As the rebuttal deadline approaches, please let us know if we have adequately addressed your concerns. If you have any remaining questions or comments about our submission, we would be happy to address them.

We appreciate your supporting positive review and thank you for all the comments and suggestions. We answer the concerns and questions raised in the following.

W1 Training time

We fully agree with the reviewer that training and inference times are important factors in assessing the relevance of this method, as well as for Physics-Informed methods in general. As you mentioned, we have provided a detailed computational time comparison for our method and the baselines in Appendix E3 for certain datasets, along with an analysis comparing our method to vanilla PINNs. While the table is too large to include in the main text, we have, following your suggestion, updated the main text to highlight the key insights from the table (section 4.4, updated version of the paper).

W2 Proof

Thanks for the comment, we agree that the initial version of the proof was somewhat confusing.

In the updated version, we have thoroughly revised this proof to better highlight the main results, their logical progression, and the connections between them (appendix B). We think that this solves the inconsistency issue mentioned in your review. Please tell us if there are still questions remaining. In particular, we have shown that the neural net optimizer leads to a condition number $\kappa(PA)$ equal to 1, which is optimal. If we consider the Poisson example, the condition number $\kappa(A)$ of the original PINNs problem being $K^4$, for $K>1$, the new condition number is greatly reduced. Note that this also directly implies that the number of steps $N^{\prime}(\epsilon)$ of our method is significantly reduced wrt to PINNs, i.e. $N^{\prime}(\epsilon)\ll N(\epsilon)$.

W3 Parameter ranges

The results presented in table 2 are averaged on several PDEs, i.e. with different parameters sampled from the ranges shown in table 1. We solved $200$ instances of PDE with varying parameters with our method, and averaged the final MSE over the solution. These PDE instances are unseen PDE during training (in terms of PDE parameters, forcing terms and/or boundary/initial conditions). Following your suggestion, we have added in appendix E1 on figure 14, an evaluation of the performance evolution of the different methods, with respect to the PDE parameters on Helmholtz equation. The results show that our method has consistent performance across the parameter distribution and is then quite robust to this variation when the baselines could only solve the PDE for only a narrow range of parameters.

If there are any remaining issues or specific questions, we’re ready to address them during the discussion period. Please let us know if there’s anything else you’d like us to clarify to convince you about our submission.

We appreciate the reviewer's feedback and support for our work, and we address their question below.

Q1

First of all, we would like to emphasize that in our method, there are two nested optimization processes: we train an optimizer (the neural solver) to accelerate the convergence of a baseline gradient (here SGD) in order to accelerate PINNs training. In this nested process, data are used for training the neural solver in the outer loop while the PDE loss is optimized in an inner loop. The use of data is then different from other algorithms like for example PINO or Physics-informed DeepONet (PI-DeepONet) that use them for solving a soft constrained problem between data and physics loss. Within the inner-loop, we did not use hard-constraints. This is a nice suggestion. We could not do the experiments during the rebuttal since this involves retraining many models but but this is an interesting direction for future work! Thank you for this suggestion.

We appreciate the reviewer's feedback and recognition of our approach's innovation and we address the raised concerns below.

W1 Related work

• comparison with ref. [4]: Firstly, we thank the reviewer for the recent reference [4] which has been added in the revised version. We have also performed new experiments with the model in [4] which appears as a new baseline in the updated table 2 (see P2INNs). As can be seen the performance of the model are close to the ones obtained with the other baselines and well below the ones obtained with our model. While running the experiments, we observed that this model also suffers from the same slow optimization schedule as the other baselines. Note that our evaluation setting is different from the one used in their paper [4]. Here we evaluate all the models for in-domain generalization, i.e. the models are trained on a set of PDE instances and tested on new instances of the PDE, with different parameters. In [4] the model is evaluated on new collocation point of the training PDEs.

• As for the positioning w.r.t. "learning to optimize" literature, you are right, this has been explored in other fields of ML. The relation with this literature is discussed in the last paragraph of our related work section in appendix A under the name of "learning to solve". We briefly introduce the literature and we refer to a thorough survey by [5]. As for its use for physics-aware ML, we found only one very recent reference that leverages meta-learning [7].

W2 Theory

We agree that our theoretical analysis relies on a simplifying linearization assumption. A proof of convergence for the actual non linear NN model would require an analysis of the non linear regime which up to our knowledge is not available. This is why we developed the analysis of the simplified linear case. Although this does not solve the non linear problem, we believe that this provides an interesting intuition and motivation for the proposed method.

The proof in the initial version of the paper was only a sketch and probably not clear enough. In the updated version, we have thoroughly revised the proof to better highlight the main results, their logical progression, and the connections between them (appendix B). In particular, we have shown that the neural net optimizer leads to a condition number $\kappa(PA)$ equal to 1, which is optimal. If we consider the Poisson example, the condition number $\kappa(A)$ of the original PINNs problem being $K^4$, for $K>1$, the new condition number is greatly reduced. Note that this also directly implies that the number of steps $N^{\prime}(\epsilon)$ of our method is significantly reduced wrt to PINNs, i.e. $N^{\prime}(\epsilon)\ll N(\epsilon)$. This is all carefully explained in the new version of the paper, appendix B.

We appreciate the reviewer's feedback and recognition of our approach's innovation and we address the raised concerns below.

W1 Proof

• Simplifying, linear assumption used in the theoretical analysis. A proof of convergence for the actual non linear NN model would require an analysis of the non linear regime which up to our knowledge is not available. This is why we developed the analysis of the simplified linear case. Although this does not solve the non linear problem, we believe that this provides an interesting intuition and motivation for the proposed method.

• Under these assumptions, the theoretical results demonstrate that $\kappa(PA)<\kappa(A)$. The proof in the initial version of the paper was only a sketch and probably not clear enough. In the updated version, we have thoroughly revised the proof to better highlight the main results, their logical progression, and the connections between them (appendix B). In particular, we have shown that the neural net optimizer leads to a condition number $\kappa(PA)$ equal to 1, which is optimal. If we consider the Poisson example, the condition number $\kappa(A)$ of the original PINNs problem being $K^4$, for $K>1$, the new condition number is greatly reduced. Note that this also directly implies that the number of steps $N^{\prime}(\epsilon)$ of our method is significantly reduced wrt to PINNs, i.e. $N^{\prime}(\epsilon)\ll N(\epsilon)$. This is all carefully explained in the new version of the paper, appendix B.

W2 Grids

The proposed method could handle as well irregular grids and different resolutions since the reconstruction loss is computed point-wise through the basis. To illustrate this, we have added a new experiment for which the training points are sampled on a irregular grid (Appendix, section E1, table 13). This experiment highlights that our approach effectively handles non-linear grids while maintaining superior performance compared to most baselines. Below is an excerpt from table 13.

Comparison of the performances when training our solver using regular or irregular and different grids for each PDE. Metrics on the test set.

W3 Training and generalization of $\mathcal{F}_{\varrho}$

• Unseen PDEs The paper targets solving parametric PDE, on a distribution of the PDE parameters including PDE coefficients, forcing terms and boundary conditions. The evaluations are performed on in-distribution, unseen equations with new parameter samples. Details for the sampling of the parameter values are provided in table 1. Our results show that the proposed method generalizes to these in-distribution unseen PDEs (Table 2, main text and Figure 14, appendix E1).

• Training of $\mathcal{F}_{\varrho}$ Following your suggestion, we have added training and testing loss plots as a function of the number of epochs for training on the Helmholtz equation with our method. These results are presented in section E5 Figures 15 and 16 of the updated manuscript and show that our method can exhibits interesting performances on the test set with only $200$ epochs of training (MSE $\approx 0.01$). Moreover, we show in appendix E1, figures 12a and 12b, a study of the performances of our method when reducing the number of available data. This experiment suggests that our model has better performance that the supervised method, even when fewer PDE solutions are available.

W4 Memory issue

We agree as indicated in the limitations that the proposed method requires a more complex training. However, this approach provides a considerable acceleration at test time compared to classical PINN optimization schemes. This acceleration is achieved through two key factors: (i) improved convergence speed at test time, and (ii) the ability to handle parametric equations, whereas vanilla PINNs, for instance, require separate training for each set of parameter values. Note that all our experiments were conducted on a single GPU with a maximum capacity of 48 GB, including training, even for 2D+time equations.

W5 Datasets choices

We acknowledge the reviewer's point about the need for competitive benchmarking. We would like to stress that we have been using a variety of representative equations, including problems challenging for PINNs - see e.g. [1, 2] - (NLRD), FNO [3] (Darcy), PDEbench [4] (advection). Our benchmarks also include non-linear datasets (NLRD), 2D as well as 2D+t datasets (Darcy and Wave) and high frequencies (pulsation going until $50$ in the Helmholtz equation) which already represent interesting benchmarks.

W6 Training and inference time cost

Following your suggestion, we have updated the main text to emphasize the key insights regarding computational cost (Section 4.4, updated version of the paper).

Thanks for the answer. Below, we provide additional insights to explain our "intuition" about the model's behavior.

• How F modifies the gradient path in the descent algorithm

Our core idea is related to the principle of "learning to solve." The optimizer F learns to adapt the gradient path dynamically for each instance of the equation. Specifically, F modifies the direction and magnitude of the stochastic gradient descent (SGD) updates in the physical loss landscape. It achieves this by utilizing solution values at various sample points to adjust the gradient, steering it toward the corresponding minimum in the mean squared error (MSE) landscape. While the descent remains within the residual physics error space, the learned optimizer provides an improved gradient direction, enhancing convergence efficiency.

• Approximation and optimal training (infinite data, optimal solution)

To further build intuition, we draw an analogy with solving linear systems. Assuming a linear approximation near the solution and ideal training conditions (i.e., infinite data and an optimal solution), the learned optimization effectively achieves ideal conditioning, where $\kappa=1$. While this linear analogy does not directly resolve nonlinear problems, it holds in the solution's vicinity and provides valuable insights into the model's local behavior.

• Illustration of convergence acceleration achieved by our method

Although we lack a general explanation for all cases, we illustrate the model's behavior using a comparison between baseline SGD and the learned optimizer in solving a Helmholtz equation. As shown in Figure 17 in the appendix, this new experiment highlights the slow convergence of PINNs compared to the significant acceleration achieved by our method. Notably, increasing the learning rate in PINNs led to divergence, further demonstrating the robustness and advantages of our approach.

As the rebuttal deadline approaches, please let us know if we have adequately addressed your concerns. If you have any remaining questions or comments about our submission, we would be happy to address them.