PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks

We thank the reviewer for the thoughtful comments on our paper.

Response to W1 & W2:

We respectfully disagree with the reviewer’s statements regarding capturing temporal dependencies with MLP-based PINNs. While it's true that MLP-based PINNs incorporate both spatial and temporal inputs, MLP-based PINNs process different inputs (i.e., batch data) independently. For standard MLP architectures, each spatio-temporal input (even with random samplings of collocation points) is treated as a distinct entity, and the model processes them separately without explicitly capturing the temporal dependencies between consecutive time steps. In contrast, PINNsFormer explicitly suggests capturing temporal dependencies within local consecutive time steps when processing each spatiotemporal input. In our experiment section, we highlight the function of PINNsFormer in explicitly capturing temporal dependencies rather than MLP-based PINNs, with clear performance benefits by empirical evaluations.

Response to W3:

No, $\Delta t$ is not fixed, it is a tunable hyperparameter. We have provided an additional ablation study regarding the different choices of $\Delta t$ in the global common response section.

Response to W4:

Yes, Spatio-Temporal Mixer in our implementation is just a one-layer MLP. The Spatio-Temporal Mixer module serves as an embedding process as in the vanilla Transformer. We show that a one-layer MLP is enough and effective to embed low-dimensional spatiotemporal information in high-dimensional spaces based on empirical evaluations.

Response to W5:

We respectfully disagree with the reviewer’s statement.

First, Positional Encoding (PE) is a typical technique used in combination with embedding methods, it serves different purposes as activation functions, including Wavelet. Also, PE and Wavelet have different mathematical expressions.

Second, Wavelet activation is not the same as Sin activation. We assign two learnable parameters $\omega_1$ and $\omega_2$ to weigh the sin and cos components in Wavelet activation. Indeed, Wavelet can be simplified to $k \sin (x+ \phi)$ for some $k$ and $\phi$ (note $\phi$ does not necessarily need equal to $\pi/2$). However, on the one hand, intuitively, compared to pure Sin activation, Wavelet introduces additional phase and magnitude information. The arbitrary learnable combination of phase and magnitude gives a larger capability than the pure Sin activation function with fixed phase and magnitude information. On the other hand, precisely, let assume $\sin (x_1) = k_1 \sin(x_2)+ k_2 \cos(x_2)$, where $x_1$ and $x_2$ are the arbitrary outputs from two independent linear layers. To reach the equality, we need have $x_2 = \arcsin(\frac{k_1}{\sqrt{k_1^2+k_2^2}}) + \arccos(\frac{\sin x}{\sqrt{k_1^2+k_2^2}})$, the translation from $x_1$ to $x_2$ involves non-linear operations, which cannot be achieved by simply add additional weights/bais to neurons. Our ablation study 1 in the global common response also empirically demonstrates the difference between Sin and Wavelet activation function.

Response to W6:

We respectfully disagree with the reviewer’s statement.

First, all our evaluations strictly follow existing prior works (convection and 1d-diffusion as [1], 1d-wave as [2], and Navier-Stokes as [3]). By evaluating these problems, we can make clear comparisons with existing baselines.

Second, we use MSE loss for all evaluations, which is widely used in PINNs. Comparing all evaluations under the same training loss metric does not lead to unfair results. If the reviewer meant for the impact of model hyperparameters, we fine-tune each model, while additionally controlling for a roughly similar number of model parameters for all baselines and PINNsFormer to avoid the effect of overparameterizations.

Third, our Navier-Stokes equation is essentially the same as the reference papers. We are using the fundamental formulation of the Navier-Stokes equation while they are using the velocity-vorticity formulation. Both formulations are mathematically equivalent, and thus, we do not believe the NS equation in our setting is simpler than the reference papers. Their showcases seem to be more complicated than ours because they are visualizing $u, v, \omega$ and we are visualizing $p$.

[1] Krishnapriyan, Aditi, et al. "Characterizing possible failure modes in physics-informed neural networks." Advances in Neural Information Processing Systems 34 (2021): 26548-26560.

[2] Wang, Sifan, Xinling Yu, and Paris Perdikaris. "When and why PINNs fail to train: A neural tangent kernel perspective." Journal of Computational Physics 449 (2022): 110768.

[3] Raissi, Maziar, Paris Perdikaris, and George E. Karniadakis. "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations." Journal of Computational physics 378 (2019): 686-707.

We thank the reviewer for the thoughtful comments on our paper.

Response to ‘Missing ablation studies on different activation functions’:

We are happy to add the ablation study on using different activation functions. We include the ablation setups, results, and conclusions in the global common response. Overall, our additional ablation shows that using Wavelet activation shows constantly better performance than ReLU, Sigmoid, and Sin activations. In particular, Sin activation shows effectiveness only in certain cases, while Wavelet can generalize all cases well.

Response to ‘Considers isotropic problems only’:

First, we strictly follow the exact same setups of PINNs as most prior works do [1,2,3], and our explorations based on existing setups and our works are meaningful. Second, the main purpose of this paper is to bring the temporal dependency idea to PINNs, with a natural adaptation of transformer architectures. Investigating anisotropic and nonlinear characteristics in physics systems with PINNsFormer is worth future explorations but yet out of our scope. For instance, a natural idea to handle direction-dependent PDEs is to introduce patches (as ViT [4] does) rather than pseudo sequences, which allows capturing higher-order dependencies than simple temporal dependencies.

Response to ‘Missing ablation study on pseudo-sequence generator’:

We are happy to add the ablation study on hyperparameters of the pseudo-sequence generator. We include the ablation setups, results, and conclusions in the global response. Please see the response above. Overall, given a mesh choice of $k$ and $\Delta t$, PINNsFormer is not sensitive to a wide range of $k$ and $\Delta t$. PINNsFormer successfully mitigates the failure modes for any combinations of $k\in [1e-2, 1e-3, 1e-4]$ and $\Delta t \in [5,7,10]$. However, we observe that either a too-large or a too-small $\Delta t$ may degrade the temporal dependencies between each time step. In the last, Increasing the pseudo-sequence length can help mitigate PINNs failure modes, but it only provides marginal benefit once our models have mitigated the failure mode/

Response to ‘Extrapolation validation should be included’:

We agree that understanding and mitigating the extrapolation (training and testing data are from different domains) failures of PINNs is a novel and challenging problem [5]. However, our paper focuses on interpolation validation as most prior PINNs works do, with showing clear benefits using PINNsFormer. The extrapolation ability of PINNsFormer is worth future explorations (maybe by adapting PINNsFormer to a forecasting model), but it is either out of our scope or not the main purpose of most PINNs works.

Response to ‘Improving the quadratic complexity’:

We agree that involving full attention will cause a $\mathcal{O}(L^2)$ complexity. However, there are existing methods such as replacing full-attention with prob-sparse self-attention [6], which reduces to $\mathcal{O}(L\log L)$ complexity. Our work aims to bring explicit temporal dependencies into PINNs, with an adaptation to Transformer as a natural solution. Improving the quadratic complexity in Transformers such as [6] is complementary future work but is currently out of our scope.

[1] Krishnapriyan, Aditi, et al. "Characterizing possible failure modes in physics-informed neural networks." Advances in Neural Information Processing Systems 34 (2021): 26548-26560.

[2] Wang, Sifan, Xinling Yu, and Paris Perdikaris. "When and why PINNs fail to train: A neural tangent kernel perspective." Journal of Computational Physics 449 (2022): 110768.

[3] Raissi, Maziar, Paris Perdikaris, and George E. Karniadakis. "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations." Journal of Computational physics 378 (2019): 686-707.

[4] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).

[5] Fesser, Lukas, Richard Qiu, and Luca D'Amico-Wong. "Understanding and Mitigating Extrapolation Failures in Physics-Informed Neural Networks." arXiv preprint arXiv:2306.09478 (2023).

[6] Zhou, Haoyi, et al. "Informer: Beyond efficient transformer for long sequence time-series forecasting." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 12. 2021.

We hope this message finds you well.

As the deadline for final evaluations is tomorrow, we would greatly appreciate any insights or suggestions you might have to our previous response, which includes two additional ablation studies and additional clarifications on you previous concerns. We would also be grateful for any consideration towards improving our current score based on potential revisions.

Thank you very much for your time and understanding.

We thank the reviewer's recognition of our work. We want to argue three points to the statements in Weakness.

(1) Similarities and Differences with Finite Difference Method:

Despite the sequential input in PINNsFormer, which can be similar to the finite difference method, they are actually different. The key idea of the finite difference method is that it discretizes the whole domain and propagates the solution globally and sequentially. PINNsFormer, instead, focuses on the temporal dependency by looking at additional collocation points near the original collocation point. In PINNsFormer, the predictions within a sequence do not have a specific sequential propagation order, but all predictions utilize temporal dependencies from each other for better predictions in parallel.

(2) Computational Cost:

Using attention mechanisms or complicated architectures can indeed raise additional computational costs. However, there are existing solutions, such as Informer [1], which uses prob-sparse attention to reduce the computation cost from $\mathcal{O} (L^2)$ to $\mathcal{O}(L\log L)$. However, the main purpose of this paper is to bring the temporal dependency idea to PINNs with a natural adaptation of transformer architectures. Investigating such mechanisms in PINNsFormer is worth future explorations but yet out of our scope.

(3) Adding Normal, Discrete PDE Solver as Baseline:

Here, we recognize the normal, discrete PDE solver as the finite element method (FEM) as the reviewer mentioned. However, FEM is typically not a baseline in the context of PINNs. Instead, research works on PINNs [2] use simulation results from FEM as the ground truth to evaluate the neural networks’ performance. Hence as such, we did not include FEM as a baseline when comparing to PINNs and their variations.

[1] Zhou, Haoyi, et al. "Informer: Beyond efficient transformer for long sequence time-series forecasting." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 12. 2021.

[2] Raissi, Maziar, Paris Perdikaris, and George E. Karniadakis. "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations." Journal of Computational physics 378 (2019): 686-707.

We thank the reviewer for the thoughtful comments for our paper.

Response to W1:

We have included the detailed implementation and hyperparameters in our paper. We include model hyperparameters in Table 4, Appendix A. We include all implementation, including demos for all experiments included in the paper (with fixed random seed for reproducibility purposes), through an anonymous GitHub repo: https://anonymous.4open.science/r/pinnsformer_iclr-4D09, as mentioned on page 7.

Response to W2:

Thanks. We are happy to provide additional ablation studies on (1) the effects of different hyperparameters in pseudo-sequence generators, and (2) the effects of using different activation functions. Please check the common response for details. Overall, our additional ablations show that (i) using Wavelet activation shows constantly better performance than ReLU, Sigmoid, and Sin activations, (ii) $k$ and $\Delta t$ show non-sensitivity to PINNsFormer's performance in general and can be chosen from a wide range.

Response to W3:

We include the computational and memory overheads in Table 5, Appendix A. Empirically, we demonstrate that the computation and memory overheads increase sublinearly as the pseudo-sequence length increases. In particular, using $k=5$ is already good enough for all experiments included, which yields tolerably approximately 3x computational cost and 2x memory cost. Theoretically, existing works such as informer [1] have been shown to reduce the complexity of attention from $\mathcal{O}(L^2)$ to $\mathcal{O}(L\log L)$ by replacing full-attention to prob-sparse attention. Investigating such mechanisms in PINNsFormer is worth future explorations but yet out of our scope.

Response to W4 & Q1:

We do use automatic differentiation rather than relying on finite difference approximations, so the advantage of PINNs is also inherited in PINNsFormer. In particular, despite a Transformer architecture to consider the temporal dependencies over multiple discrete time steps, the gradient calculation, as specified in Equation 5, is still executed point-wisely through automatic differentiation. We are happy to add more detailed explanations in the final version. The investigation of different $\Delta t$s is included in Response to W2.

Response to Q2:

We thank the reviewer’s suggestion on causal-attention transformers. However, the main purpose of this paper is to bring the temporal dependency idea to PINNs, with a natural adaptation of transformer architectures. Investigating more advanced transformer-based architectures is worth future explorations but yet out of our scope.

Response to Q3:

The adaptive sampling strategy can be easily incorporated into PINNsFormer, as we have demonstrated the flexibility of incorporating PINNsFormer with existing training schemes in Section 4.3. For instance, the mentioned paper uses adaptive samplings according to the computed risks. One can also incorporate this to PINNsFormer by (1) input collocation point (x,t), (2) output sequential output $[ \hat{u}(x,t+i\Delta t)]_{i=0}^{k-1}$, (3) compute the weight based on either the sum of the risk of all elements of the sequence or the risk of the first element of the sequence.

[1] Zhou, Haoyi, et al. "Informer: Beyond efficient transformer for long sequence time-series forecasting." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 12. 2021.

We hope this message finds you well.

As the deadline for final evaluations is tomorrow, we would greatly appreciate any insights or suggestions you might have to our previous response, which includes two additional ablation studies and additional clarifications on you previous concerns. We would also be grateful for any consideration towards improving our current score based on potential revisions.

Thank you very much for your time and understanding.