HyResPINNs: Adaptive Hybrid Residual Networks for Learning Optimal Combinations of Neural and RBF Components for Physics-Informed Modeling

Regarding Question # 3

We thank the reviewer for raising this question. The Wendland $C^4$ kernel was chosen due to its compact support, smoothness, and computational efficiency, making it well-suited for the hybrid architecture. Compactly supported kernels, like Wendland $C^4$, result in sparse kernel matrices, which improve scalability and reduce computational overhead, especially for high-dimensional problems. Specifically, the kernel value is zero for points outside a specified radius, $\tau$ around the kernel center. This compact support property means that most entries in the kernel matrix are zero, as interactions only occur between points within the support radius. Consequently, the resulting sparse kernel matrices require less memory for storage and enable the use of efficient sparse matrix solvers, significantly reducing the computational complexity compared to dense kernel methods. Additionally, the $C^4$ smoothness ensures sufficient differentiability for solving PDEs that require higher-order derivatives.

While the Wendland $C^4$ kernel provides a balanced trade-off between computational efficiency and smoothness, we acknowledge that this choice is one of several possibilities. Other kernel functions, such as Gaussian or Matérn kernels, could also be explored, depending on the problem requirements. For instance, Gaussian kernels are widely used for their universal approximation properties, but they lack compact support, leading to dense kernel matrices that scale poorly with the problem size. Investigating the performance of alternative kernel functions tailored to specific PDEs or problem domains is a promising direction for future work. We appreciate the reviewer’s suggestion to clarify this point, and we will emphasize this discussion in the revised manuscript to better highlight the rationale for our choice and its implications.

We thank the reviewer for their comments. Below, we address each question and comment individually.

Weakness # 1

Thank you for this observation. We appreciate the need to explicitly report computational cost for the baselines, as these metrics are crucial for understanding the practical trade-offs of different approaches. To address this concern, please see our response above to Reviewer ba3x. Regarding the similarity between PirateNet and HyResPINN in terms of training time (as shown in Figure 6), we agree that the differences are less pronounced during certain training regimes. However, we note that our model consistently achieves superior accuracy for a given computational budget, as demonstrated by the faster convergence in mean relative L2 error.

Weakness # 2

We appreciate the reviewer’s comments suggesting the inclusion of additional PDE examples. While the selected problems (e.g., Allen-Cahn and Darcy Flow) were chosen to evaluate distinct aspects of our method, such as handling nonlinear dynamics and varying boundary conditions, we recognize that additional examples would further strengthen the evaluation. If accepted, we plan to expand our experiments to include additional PDEs.

Weakness # 3

We appreciate the reviewer's concern regarding the technical novelty of our approach. While it is true that our work builds on the residual architecture used in PirateNet, the integration of RBF networks is not simply a substitution but rather a concrete enhancement that introduces new capabilities and performance improvements.

Specifically, by leveraging RBF networks, our method achieves a more refined representation of sharp transitions in solutions, as illustrated in Figure 4. RBF kernels' localized and adaptive nature enables the hybrid architecture to balance smooth and sharp features, a capability that PirateNets and the other baselines struggle to achieve. This improvement is further validated by the consistently lower errors across a wide range of PDEs, as demonstrated in Table 1. Moreover, using RBF networks within the residual framework enriches the expressive power of the model. Unlike traditional neural networks, RBF kernels provide localized basis functions that adaptively capture fine-grained solution structures. This integration allows for a more flexible and efficient representation of PDE solutions, particularly for problems with complex features.

While existing architectures inspire our approach, combining neural networks with RBF-based methods for PDE solving represents a novel contribution. This hybridization bridges two previously distinct paradigms and opens new avenues for extending residual-based architectures.

Regarding Question # 1

We appreciate the reviewer’s question regarding whether $\alpha^{(l)}$ should depend on the input $\mathbf{x}$. In our current implementation, $\alpha^{(l)}$ is a trainable scalar shared across the domain. This design choice simplifies the model and reduces the number of parameters while still capturing a wide range of solutions, as shown in our experiments.

That said, making $\alpha^{(l)}$ dependent on $\mathbf{x}$ could offer additional flexibility, enabling the model to adapt the contributions of the RBF and neural network components based on local solution features. For example, in regions with sharp transitions, the RBF component might dominate, while in smoother regions, the neural network could take priority. Exploring this idea is a promising direction for future work, and we thank the reviewer for raising it.

Regarding Question # 2

We thank the reviewer for highlighting this notation issue. We have renamed the kernel description in the revised manuscript to distinguish it from the sigmoid function.

Hello,

I just wanted to confirm the authors have not made any changes/replies to their submission, is that correct? As the discussion period is set to end today, there might not be enough time for a timely response from reviewers, should the authors decide to reply to our reviews.

We thank the reviewer for their comments. Below, we address each question individually.

Including More PDEs In Experiments

We appreciate the reviewer’s comments suggesting the inclusion of additional PDE examples. While the selected problems (e.g., Allen-Cahn and Darcy Flow) were chosen to evaluate distinct aspects of our method, such as handling nonlinear dynamics and varying boundary conditions, we recognize that additional examples would further strengthen the evaluation. If accepted, we plan to expand our experiments to include additional PDEs.

Reconsidering Notion of "training size"

We thank the reviewer for highlighting this aspect of PINN training and for pointing out the advantages of sampling independent random collocation points at each iteration. We agree that this approach often leads to more accurate and robust performance, and we have employed this strategy for the Allen-Cahn experiments, as detailed in the Appendix. Specifically, we mention that collocation points were randomly sampled during each training iteration for this experiment, utilizing the mesh-free nature of PINNs.

For other experiments, such as Darcy flow, we used fixed collocation points instead, as this setup is more commonly employed for comparison against baseline methods in the existing literature. This choice also allowed us to explore the impact of training set size on the performance of various architectures, which remains a common experimental protocol in the PINN community. While we recognize that this approach may not leverage the full flexibility of PINNs, it provides meaningful comparisons in contexts where fixed-point setups are standard.

Clarity on Formulation of Residual Connections

We thank the reviewer for their feedback and for pointing out areas where further clarity is needed regarding the formulation of the residual connections and initialization of parameters. In our current implementation, $\beta^l$ is initialized to 10, ensuring that $\phi(\beta^l) \approx 1$. Similarly, $\alpha^l$ is initialized to 0, such that $\phi(\alpha^l) = 0.5$, ensuring an equal balance between the contributions of the RBF and NN components within each block at the start of training. While the $\alpha^l$ values are regularized during training, the $\beta^l$ parameters are constrained only by the sigmoid function $\phi$, ensuring their values remain within $(0, 1)$ for stable training. By initializing $\phi(\beta^l) \approx 1$, the residual connections initially fully pass the outputs from previous blocks. Similarly, the equal weighting of $\phi(\alpha^l) = 0.5$ ensures neither component dominates prematurely. We will add these clarifications to the revised manuscript and further discuss the roles of $\beta^l$ and $\alpha^l$ and how their adaptive modulation contributes to the hybrid model's flexibility and performance.

Other RBF Kernels

We thank the reviewer for suggesting exploring alternative kernels in the RBF blocks of HyResPINNs. We agree that investigating the effect of other kernels, such as Gaussian or Matérn kernels, could provide valuable insights into the flexibility and robustness of the hybrid architecture. If the paper is accepted, we will include an ablation study in the appendix, evaluating the performance of HyResPINNs with different kernels on a representative PDE problem. This experiment will allow us to assess the trade-offs between kernel choices regarding accuracy, efficiency, and adaptability to varying solution characteristics. We believe this addition will enrich the paper and further clarify the impact of the kernel selection on the proposed framework.

Reporting Computation Time

We appreciate the reviewer’s suggestion to provide more details on computational time and the hardware used for experiments. In the manuscript, we mentioned the training hardware (NVIDIA A100 GPU running CentOS 7.2) in the "Experimental Setup" section. Additionally, we included a brief computational cost analysis in Figure 6 (see our response to reviewer ba3x). However, we acknowledge that the explicit reporting of training times for each method and experiment was not fully detailed. To address this, we have now included the training time for each method in the table above (see our response to reviewer ba3x). This table compares wall-clock training times for all methods.

Code as supplementary material

We fully agree that providing code is essential for reproducibility and for gaining a better understanding of the method. We plan to release the complete codebase for our methods upon acceptance of the paper. This code will include scripts for training and evaluating the proposed HyResPINN architecture and reproducing all experimental results presented in the manuscript. The release will ensure that readers and researchers can evaluate the execution of experiments and extend the work easily. We have added a note in the "Experimental Setup" section clarifying our commitment to making the code publicly available upon publication.

RBF Netowork References

We thank the reviewer for pointing out the lack of references for Radial Basis Function (RBF) networks in the Introduction and Section 2.2. We agree that including relevant references would strengthen the context and situate our work more effectively within the broader literature. To address this, we have added key references to RBF networks in the introduction (see Line 062), including (Bai et al. 2023) and (Fu et al. 2024), who use physics-informed RBF networks to solve PDEs.

Finally, we thank the reviewers for their careful reading of our manuscript and for pointing out the grammatical, typographical, and notational errors. In the revised manuscript, we have updated the notation in Equations (1) and (10), fixed the typo in the diagram, and fixed the reference pointers in the Appendix. We will additionally add a footnote acknowledging the ill-posed nature of the problem and referencing relevant papers that address it with zero Dirichlet boundary conditions.

We thank the reviewer for their comments. Below we address each weakness individually.

Regarding the computational complexity of the proposed approach.

Thank you for highlighting the importance of discussing computational complexity. In the original submission, we included an analysis of computational cost in Figure 6, where the right plot explicitly compares the mean wall-clock training time and error across different methods, including our proposed approach (HyResPINNs). As shown, while HyResPINN has a slightly higher (overall) training cost, it outperforms baseline methods in terms of accuracy. Specifically, the vertical lines in the figure indicate how long each model trained until achieving a relative L2 error of 10^-2. The figure shows that ResPINNs, ExpertPINNs, and HyResPINNs (ours) achieve this threshold at similar wall-clock times, while PirateNets took the longest. This trade-off between training time and accuracy highlights the robustness and efficiency of our approach to solving complex PDEs.

However, to further address your concerns, the table below includes a more detailed breakdown of computational complexity. Specifically, this table analyzes the wall-clock training time (in minutes) of HyResPINNs compared to each baseline approach corresponding to the best accuracy results presented in Table 1.

Regarding PDE benchmarks

We appreciate the reviewer’s comments regarding the limited number of benchmark PDEs in our current experiments. While the selected problems (e.g., Allen-Cahn and Darcy Flow) were chosen to evaluate distinct aspects of our method, such as handling nonlinear dynamics and varying boundary conditions, we recognize that additional examples would further strengthen the evaluation. If accepted, we plan to expand our experiments to include additional PDEs.

RBF kernel explanation.

We appreciate the reviewer's observation regarding Figure 3 and would like to clarify the relationship between kernel smoothness and the ability to approximate sharp transitions. While Figure 3 visualizes the learned RBF kernels, their smooth appearance reflects the inherent properties of individual RBF kernels rather than the overall behavior of the hybrid model.

Specifically, when using Wendland kernels—a compactly supported RBF kernel—the smoothness is localized within the kernel's support region, and the kernel transitions sharply to zero at the boundary of its support. This compact support introduces non-smooth behavior at the edges while maintaining smoothness within the support region. The hybrid model benefits from the locality in the Wendland kernels, which ensures that each kernel focuses on a specific region of the solution, while the neural network component provides the flexibility to adjust to global features. Together, each component enables the hybrid model to balance smooth behavior and sharp transitions effectively.

Further, to directly address this concern, we refer the reviewer to Figure 4, which provides evidence of the HyResPINN's ability to handle sharp transitions. Specifically, the absolute error plots in the second row of Figure 4 illustrate that HyResPINN consistently achieves lower errors near sharp transitions compared to both the standard PINN and the RBF network (RBFNet). These results highlight the hybrid model's effectiveness in capturing sharp features where other methods falter. The bottom row of Figure 4 shows that HyResPINN accurately captures smooth and sharp features in the Allen-Cahn solution across all time steps, aligning closely with the exact solution (green). In contrast, the standard PINN struggles to resolve sharp features, resulting in large errors.

Are the centers and coefficients trained in each RBF-NN?

In our proposed adaptive RBF kernel approach, the scaling parameters ($\tau_i$) of the Wendland kernel, as well as the kernel weights ($W$ in Equation 12), are trainable model parameters. These parameters are optimized via gradient descent alongside all other network parameters during training.

We thank the reviewers for their careful reading of our manuscript and for pointing out the grammatical and notational errors. In the revised manuscript, we have updated the notation in Equation (4), fixed the grammatical errors, and fixed the reference pointers in the Appendix.