Enhancing Physics-Informed Neural Networks Through Feature Engineering

While the key idea of using feature engineering for improving the performance of PINNs has merit, I have several major concerns about the current presentation and experimentation presented in the manuscript:

1. The experimental comparisons rely heavily on vanilla MLP networks, which are known to be a very weak baseline for PINNs. The absence of comparisons against state-of-the-art methods like PirateNets [1] makes it difficult to assess the true significance of SAFE-NET's contributions. A more compelling evaluation would combine SAFE-NET's feature embeddings and optimization protocol with architectures like PirateNets to determine if they can achieve state-of-the-art results. This would provide stronger evidence for the generalizability of the proposed approach.

2. The paper's evaluation is limited to PDEs with smooth solutions, which represent relatively simple benchmarks in the current PINN literature. While these problems do highlight basic PINN training challenges, recent work has introduced substantially more challenging benchmarks where competitive results have been achieved by other methods [1,2]. The authors should extend their evaluation to these more demanding scenarios, such as problems with discontinuities, sharp gradients, or multi-scale features, to properly demonstrate SAFE-NET's capabilities against the current state-of-the-art. More concretely, I suggest that you consider evaluating your method against SOTA PINNs methods likePirateNets on the following benchmarks: KS, Gray-Scott and Ginzburg-Landau (see [1,2]).

3. The feature map selection process appears to lack a rigorous foundation. The authors switch between different feature types (e.g., Fourier embeddings for wave equations, logarithmic and polynomial features for reaction PDEs) without providing clear mathematical justification for these choices. A significant enhancement would be developing a systematic methodology for selecting feature maps based on PDE properties. For instance, with linear PDEs, one would expect optimal feature maps to correspond to PDE operator eigenfunctions. Without such theoretical grounding, the feature selection process remains largely heuristic and it is hard for me to see it achieve practical impact.

4. The (Adam + L-BFGS)² optimization protocol, while showing some empirical improvements, lacks theoretical justification. The paper would be strengthened by providing mathematical intuition or analysis explaining why this particular sequence of optimizers is effective. The current results, while promising, are limited to simple PDEs and weak baselines (see comments above). This is particularly concerning given that some state-of-the-art PINN results have been achieved using standard Adam optimization with default parameters [1,2], raising questions about the broader value of the proposed optimization strategy.

5. The authors' assertion that inadequate input representations are the primary challenge in PINN training overlooks other crucial factors identified in recent literature. Specifically, research has shown that derivative pathologies in deep MLP networks (independent of feature engineering) contribute significantly to optimization difficulties [1]. By focusing primarily on feature engineering, the paper may underestimate the importance of other architectural considerations, initialization schemes, and optimization challenges in PINN training. I doubt that feature engineering alone is the recipe to resolve the issues that PINNs face in challenging PDE settings.

6. The paper's discussion of feature map design based on dominant frequencies needs stronger validation on more challenging time-dependent problems. The Kuramoto-Sivashinsky equation (see comments above) would be a particularly valuable benchmark, as its solution exhibits time-varying dominant frequencies characteristic of chaotic systems. The current benchmarks all have relatively stable frequency characteristics, making it unclear how SAFE-NET's feature engineering strategies would adapt to problems where the relevant frequencies evolve significantly over time. Success with the KS equation would require either demonstrating that the proposed feature selection methods can capture temporally varying frequency structures, or acknowledging this limitation and proposing potential extensions. This benchmark would provide important insight into both the capabilities and limitations of the SAFE-NET framework for more complex time-dependent PDEs.

7. The paper's claims about SAFE-NET's novelty and differentiation from existing approaches in Section 3 are overstated. While the authors present SAFE-NET as significantly different from existing methods, similar ideas about engineered features in PINNs have been explored in previous work. For example, Delta-PINNs [3] and other approaches [4,5] have proposed tailored feature engineering strategies for specific PDEs. Rather than emphasizing novelty, the paper would be stronger if it positioned its contribution as a systematic exploration of an important open question: how to determine effective feature maps for different PDE systems. This reframing would better align with the paper's actual contributions while acknowledging its place in a broader line of research on feature engineering in PINNs.

[1] Wang, S., Li, B., Chen, Y., & Perdikaris, P. (2024). PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks. arXiv preprint arXiv:2402.00326.

[2] Wang, S., Sankaran, S., Wang, H., & Perdikaris, P. (2023). An expert's guide to training physics-informed neural networks. arXiv preprint arXiv:2308.08468.

[3] Costabal, F. S., Pezzuto, S., & Perdikaris, P. (2024). Δ-PINNs: physics-informed neural networks on complex geometries. Engineering Applications of Artificial Intelligence, 127, 107324.

[4] Hu, W. F., Lin, T. S., & Lai, M. C. (2022). A discontinuity capturing shallow neural network for elliptic interface problems. Journal of Computational Physics, 469, 111576.

[5] Wu, J., Wang, S. F., & Perdikaris, P. (2023). A dive into spectral inference networks: improved algorithms for self-supervised learning of continuous spectral representations. Applied Mathematics and Mechanics, 44(7), 1199-1224.

W1) There is no mention of the types of scenarios, equations etc. where the method fails (and where it works) such types of equations, regimes, initial or boundary conditions. The method does work on the canonical examples used in the experiments, however how does the feature engineering help with the learning is not clear. Look at the question on the Logarithmic and Polynomial features (Q1)

W2) It has been mentioned multiple times that the network is a single layer network, however there is no clarity on the size of input, output and the hidden layer. There is a mention of the parameters being reduced from 5300 to 600, however it would be beneficial if all the sizes and corresponding layer wise parameters are mentioned clearly. (As this is one of the main contributions of the paper, reducing the model complexity and size)

W3) The proposed optimizer doesn’t have strong theoretical guarantees. See question below (Q3)

W4) This work doesn’t study complex high dimensional equations which are very standard in PINN/SciML literature such as Navier-Stokes (Kovasznay Flow etc.). What challenges they anticipate in applying SAFE-NET to such scenarios or scaling them?

Presentation/Typos/Corrections

Why is section 2 the Problem Setup and Section 3 as related work? (I believe this breaks the flow of the work and the sections could be interchanged). I also think the details about learning rates, discretization domains, mesh, points etc could have been included in the section 5 on experiments.

In section A.4 line 718, what is being referred by $K$ ?

In appendix A.2 Wave Equation, the parameter $\beta$ is not mentioned correctly, I believe the initial condition on line 682 should be $sin(\beta \pi x)$

In Appendix B, is there a subsequent B.2, B.3 section as the section is marked as B.1 ?

The paper lacks clarity in some explanations, making it challenging to fully understand key details. Specific areas needing improvement are:

1. Data Processing: The paper does not clearly explain how data is processed and fed into the model. It appears that Fourier transforms are used to focus on specific frequencies within the data, which may imply that SAFE-NET selectively captures frequency information. Could authors explain the exact feature engineering process, including any frequency selection criteria, and how the selected features are incorporated into the network architecture?. And could the authors clarify if the model indeed restricts itself to certain frequency bands, and if so, how this impacts performance and generalization?

2. Hyperparameter Sensitivity: The model appears to involve numerous hyperparameters, not only from the PINN setup but also from the feature engineering strategy. This complexity could suggest sensitivity to hyperparameter choices, yet the paper does not include ablation studies or analyses of hyperparameter robustness. Assessing how different parameter configurations impact model performance would provide a clearer understanding of SAFE-NET’s flexibility and reliability.

3. Optimizer Explanation: The use of Adam and L-BFGS optimizers, while promising, would benefit from further explanation. Though widely used, the rationale behind combining these optimizers specifically within SAFE-NET’s framework remains unclear. The paper should stand on its own in terms of clarity And The paper would be strengthened by a discussion of why neither optimizer alone suffices, as well as insights into how their combination uniquely contributes to performance improvements. This explanation would enhance the paper’s self-containment, allowing readers to appreciate the methodological choices without needing prior knowledge of these optimizers.