Preconditioning for Physics-Informed Neural Networks

Dear reviewer zaqP,

Thank you for your valuable feedback. Here are our responses to your concerns, which consist of two parts: Part I (Q1-2), Part II (Q3).

Q1: Novelty of condition number.

We appreciate your insights on the long-standing study of condition numbers in numerical analysis. While the general concept of condition numbers is indeed well-established, its application in specific domains, such as PINNs, requires nuanced adaptation and innovation. Our work extends this concept to the specific context of PINNs, where broad, abstract definitions are implemented to gain domain-specific insights.

The 1976 paper you referenced, while seminal, primarily discusses mathematical properties in a broad context and does not delve into the impact of condition numbers on neural network training, particularly in the context of PINNs. Our paper, in contrast, introduces the use of condition numbers as a diagnostic tool specifically for evaluating and improving the training dynamics of PINNs. We substantiate this approach both theoretically and empirically (refer to Section 5.2 and 5.3), demonstrating its influence on accuracy and convergence. Our empirical results (see Table 2 and Figure 3(c)) illustrate that without adequate preconditioning, even advanced PINN methods can fail, underscoring the significance and novelty of our findings in the evolution of PINN research.

We agree that certain concepts have roots in earlier studies, but our application to PINNs represents a novel and significant advancement. The efficiency of PINNs in solving inverse problems, which we address in Q2, further emphasizes the unique contributions and practical implications of our work for the PINN community.

Q2: Scalability to high-dimensional problems.

We appreciate the reviewer's concern regarding the scalability of our method to high-dimensional PDEs. As noted in Section 5.3 of our paper, our current approach is indeed more suited to physical phenomena in spaces with dimensions $d\le 4$. This focus aligns with many real-world applications, particularly in macroscopic physical contexts. Besides, Figure 3(b) demonstrates superior scaling with respect to mesh size compared to traditional methods like FEM, suggesting potential advantages in larger problem settings.

Furthermore, we highlight the effectiveness of our approach in inverse problems [1,2], a significant domain in science and engineering where PINNs exhibit notable advantages over traditional methods. We have additionally studied two inverse problems from the benchmark [3]. The results, as shown in the table below, indicate that our method not only surpasses state-of-the-art PINN baselines but also outperforms the adjoint method in terms of speed and accuracy. This is particularly relevant as numerical methods often struggle with noise sensitivity in inverse problems.

We believe these results strongly support the applicability and efficacy of our method in relevant practical scenarios.

[1] Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021). Physics-informed machine learning. Nature Reviews Physics, 3(6), 422-440.

[2] Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378, 686-707.

[3] Hao, Z., Yao, J., Su, C., Su, H., Wang, Z., Lu, F., ... & Zhu, J. (2023). PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs. arXiv preprint arXiv:2306.08827.

Q3: Comparison to related work.

We appreciate the opportunity to compare our work with the referenced paper [1]. While both studies aim to enhance PINNs, there are key differences that highlight the novelty and value of our approach:

1. Theory vs. Heuristics: Our paper focuses on defining and analyzing the condition number in the context of PINNs, providing a theoretical framework for understanding and resolving training pathologies (refer to Definition 3.1, Corollary 3.5, and Theorem 3.6). In contrast, [1] heuristically enhances the L-BFGS optimizer without a theoretical discussion on condition numbers. Our work thus offers a theory-driven perspective, contributing to a deeper understanding of PINNs.
2. Integration with PINNs: Our method is intricately linked with the properties of PINNs, reflecting a direct discussion on Physics-Informed Machine Learning. We explore the condition number from a PDE standpoint, making our analysis and conclusions highly relevant to PINNs. The approach in [1], while valuable, appears more aligned with general optimization techniques.
3. Comprehensive Experiments: Our experimental setup is extensive, involving 18 PDEs and 2 inverse problems, with comparisons against over 10 PINN methods. This breadth ensures robust validation of our findings. While [1] provides valuable insights, their experimental scope is more limited (4 specific PDEs), lacking in areas like inverse problems and diverse PDE challenges compared to ours (e.g., discontinuity, complex geometry, refer to Table 1 in our paper).
4. Scalability and Applicability: Our mesh-based approach to enforcing boundary conditions is adaptable to complex geometries, an aspect not fully addressed in [1] (no such experiment is included). This adaptability enhances the practicality of our method in real-world applications with intricate geometrical considerations.
5. Open Access and Reproducibility: We have made our code and implementation details available, fostering transparency and facilitating further research. It could strengthen their findings and enable potential integration with our work for enhanced analysis if [1]'s code is available in the future.

In conclusion, while [1] offers valuable insights for optimizer improvement, our work complements theirs by providing a theoretical foundation and broader experimental validation for PINNs. We believe these differences underscore the unique contributions of our research.

[1] Alena Kopaničáková, Hardik Kothari, George Em Karniadakis, Rolf Krause, Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies, arXiv:2306.17648

Dear Reviewer,

As the discussion phase of the review process is drawing to a close, we wish to respectfully seek your input on the revisions we have implemented in response to your insightful feedback. In our commitment to advancing the quality of our research, we have conducted additional experiments on the inverse problem. These new experiments are designed to explicitly demonstrate the superiority of our approach over both traditional numerical methods and other PINN baselines. Furthermore, we have performed a thorough comparison with the related work you referenced, highlighting the distinct and novel aspects of our study.

Your feedback has been instrumental in guiding these improvements, and we highly value your expertise in this field. We are eager to hear your thoughts on these latest revisions. Please know that we are fully open to engaging in further discussions and making any additional improvements as needed.

We look forward to your valuable insights and guidance.

Warm regards,

Authors

Clarification on Our Main Contribution

We are thankful for your recognition of the effectiveness of our preconditioning method in inverse problems. Our contribution goes beyond this, as demonstrated in our experimental results (Table 2). Our method not only achieves state-of-the-art results in most tested problems but also demonstrates superior performance, reducing errors by an order of magnitude or being the only method to significantly reduce errors below 100% in 7 problems. This practical utility underscores the novel and impactful nature of our work, contributing meaningfully to advancements in the field.

Addressing the Concern on Convexity Assumption

We understand your concern regarding the assumption of convexity in our analysis. It's true that global convergence guarantees for neural network training, a typically non-convex optimization problem, are elusive. As highlighted in our research and Table 2, divergence in some PINN problems (e.g., Wave2d-MS) is a real challenge.

However, our focus on local convergence is underpinned by the rationale that in the neighborhood of a minimum, the loss function can be approximated effectively by its truncated Taylor expansion, implying local convexity. This approach is a widely accepted practice in non-convex optimization literature, as exemplified in [1] (Page 14).

Introduction of an Additional Theorem

In response to your valuable feedback, we have expanded our theoretical analysis. We introduce a new theorem in the revised manuscript (now in Theorem 3.6) that investigates the global convergence of PINNs without relying on the convexity assumption. This analysis utilizes the neural tangent kernel (NTK) framework to establish a link between the condition number and the rate of global convergence under NTK's assumptions. We believe this addition addresses your concerns and enriches the paper significantly. The detailed proof is presented in Appendix A.7. We invite you to review this new theorem and look forward to your thoughts.

[1] https://www.princeton.edu/~aaa/Public/Teaching/ORF363_COS323/F14/ORF363_COS323_F14_Lec8.pdf

Q4: Question about Eq. (11).

We appreciate the reviewer’s attention to the details in our manuscript and apologize for any confusion caused by the initial presentation of Eq. (11). In response to your concerns, we have provided a detailed and rigorous mathematical derivation in Appendix B.1 of the revised manuscript.

The statement that "simply replaced u_theta with the target u" is factually incorrect. If such a substitution were made in Eq. (4) of Definition 3.1, it would lead to a meaningless "0/0" situation.

We encourage the reviewer to examine the revised section in Appendix B.1 for a comprehensive understanding of the derivation and its implications. We are confident that this additional information will clarify any misunderstandings and demonstrate the correctness of our approach.

Q5: Many other flaws.

Thank you for your review. We note your comment about 'many more flaws' in our paper. To ensure we can address these effectively, could you please provide specific details about these additional concerns? Your insights are crucial for us to improve and refine our manuscript. We appreciate your time and expertise in guiding our revisions.

Dear reviewer 8B8P,

Thank you for your insightful questions regarding several important problems. Here are our responses to address your concerns, which consist of two parts: Part I (Q1-3) and Part II (Q4-5).

Q1: Definition of Condition Number

"The 'condition number' you define looks nothing like a condition number" is a factual error. According to [1], the general condition number is defined to be:

To state the first one we need to define the relative condition number of the mapping $g$ at $x$ as

$$\kappa_{\text{rel}}(g,x)\equiv \limsup_{\delta x \rightarrow 0} \frac{\\| g(x+\delta x)-g(x) \\|/\\| g(x) \\|}{\\| \delta x \\| / \\| x \\|}$$

Our definition is not an arbitrary creation but rather a thoughtful extension of the general condition number concept to the specific domain of Physics-Informed Neural Networks (PINNs). Delving into Definition 3.1, we can see our definition exactly matches the requirement: compare $\delta x$ with $\delta f$ and $g(x+\delta x)-g(x)$ with $\delta u$. Besides, we clarify that supremum is not taken on any dependent variable but on the trainable parameters $\boldsymbol{\theta}$. We apologize that in Definition 3.1, we have neglected $\boldsymbol{\theta}\in \Theta$ for simplicity. To ensure there is no ambiguity, we have added this notation in our revised manuscript.

While the general definition of the condition number provides a broad framework, it is often necessary to tailor this concept to specific domains for practical application. This practice is not uncommon, as seen in matrix theory and root-finding algorithms. Our paper introduces a specialized definition of the condition number for PINNs, aimed at addressing and rectifying their unique training pathologies. This approach is both theoretically and empirically validated in Sections 5.2 and 5.3 of our paper, demonstrating its relevance and effectiveness in improving the accuracy and convergence of PINNs. We assert that the condition number's ill-conditioned properties in PDEs are a critical factor affecting the performance of PINNs. This is substantiated by our comparative analyses (refer to Table 2 and Figure 3(c)) and highlights the significance of our contribution to the PINN community. We believe that our findings offer a valuable direction for future research in this field.

[1] Demmel, J. W. (1987). On condition numbers and the distance to the nearest ill-posed problem. Numerische Mathematik, 51, 251-289.

Q2: The central theorem 3.6, ..., looks nothing like a result one would obtain in the linear case.

"Theorem 3.6 should look like a result in the linear case" is factually incorrect. The neural network has non-linear activation functions, whose optimization should not have the same convergence as the linear optimization. Specifically, linear optimization has well-established and predictable convergence conditions while the global convergence of NNs cannot be assured. Furthermore, Figure 2(c) demonstrates that as the condition number decreases, there is a corresponding acceleration in convergence, empirically supporting the validity of our theorem.

Q3: "Training PINNs with a Preconditioner" they do not use a NN.

"Training PINNs with a Preconditioner they do not use a NN" is a factual error. In Line 7 of Algorithm 1, we have explicitly declared the use of the neural network. Furthermore, in Appendix D.1, we provide detailed information regarding the architecture and hyperparameters of the neural network used, which further affirms its integral role in our work. Our model is a physics-informed neural network with a preconditioned discretized loss, fundamentally differentiating it from traditional FEM approaches.

To further clarify, $u_{\\boldsymbol{\\theta}}$ is a neural network that takes a coordinate $\\boldsymbol{x}$ as input and outputs a scalar for prediction and the vector $\\boldsymbol{u}_{\\boldsymbol{\\theta}}$ is not a static vector but the neural network outputs at the grid locations:

Then, we incorporate the network outputs $\\boldsymbol{u}_{\\boldsymbol{\\theta}}$ into the preconditioned loss function (refer to Eq. (12)).

We do not optimize the vector $\\boldsymbol{u}_{\\boldsymbol{\\theta}}$ directly but the trainable parameters $\\boldsymbol{\theta}$ of the neural network.

Q1: Supremum over dependent variable.

We believe your concern has been addressed since the assumption is explicitly mentioned in Definition 3.1.

Q2: Definition of condition number.

It seems that we have reached a consensus: the condition number we defined meets the definition of condition number. The main disagreement now is whether the problem output should be the predicted solution or the optimal network parameters. We think it should be the former (our current choice), based on the following reasons:

1. The point is that the optimal parameter $\boldsymbol{\theta}^*$ is likely to be non-unique as the universal approximation theorem [1] does not ensure the uniqueness. We cannot make the definition well-defined without the uniqueness of $\boldsymbol{\theta}^*$. Specifically, there could exist another $\boldsymbol{\theta}'$ such that $\\| \boldsymbol{\theta}' - \boldsymbol{\theta}^* \\|$ is non-zero but $u_{\boldsymbol{\theta}'}$ already approaches $u$.

2. What we really care about in the PINN problem is how well the neural network $u_{\boldsymbol{\theta}}$ approximates the solution $u$ instead of how close between $\boldsymbol{\theta}$ and $\boldsymbol{\theta}^*$ is. As discussed in [2], in different random trials, we are likely to obtain different parameters that are far from each other but all well approximate $u$. It is clearly unreasonable to measure $\\| \boldsymbol{\theta} - \boldsymbol{\theta}^* \\|$ in this case.

3. Empirically, in Figure 2(b) and 2(c), we can find that the current condition number can significantly affect the accuracy and convergence of PINNs. Furthermore, our ablation study shows that the NN architecture plays a minimal role in the performance of PINNs.

   Different number of hidden neural neurons in each layer (the number of hidden layers is 5; the problem is Poisson2d-MS; 3 random trails):

   Metric (mean ± std)	$32$	$64$	$128$	$256$	$512$
   MAE	8.42e-2 ± 3.77e-4	8.38e-2 ± 2.36e-4	8.60e-2 ± 3.07e-3	8.84e-2 ± 2.05e-3	8.49e-2 ± 8.01e-4
   MSE	2.72e-2 ± 1.89e-4	2.73e-2 ± 2.94e-4	2.80e-2 ± 1.01e-3	2.90e-2 ± 8.38e-4	2.75e-2 ± 1.89e-4
   L1RE	4.75e-2 ± 2.16e-4	4.73e-2 ± 1.41e-4	4.85e-2 ± 1.75e-3	4.99e-2 ± 1.13e-3	4.79e-2 ± 4.50e-4
   L2RE	6.36e-2 ± 2.36e-4	6.36e-2 ± 3.30e-4	6.44e-2 ± 1.16e-3	6.56e-2 ± 9.63e-4	6.38e-2 ± 2.36e-4

   Different number of hidden layers (the number of hidden neural neurons in each layer is 128; the problem is Poisson2d-MS; 3 random trails):

   Metric (mean ± std)	$3$	$4$	$5$	$6$	$7$
   MAE	8.39e-2 ± 6.55e-4	8.37e-2 ± 8.29e-4	8.84e-2 ± 2.05e-3	8.21e-2 ± 4.64e-4	8.43e-2 ± 4.50e-4
   MSE	2.72e-2 ± 1.41e-4	2.70e-2 ± 2.87e-4	2.90e-2 ± 8.38e-4	2.56e-2 ± 2.36e-4	2.73e-2 ± 4.71e-5
   L1RE	4.74e-2 ± 3.68e-4	4.72e-2 ± 4.64e-4	4.99e-2 ± 1.13e-3	4.63e-2 ± 2.49e-4	4.75e-2 ± 2.49e-4
   L2RE	6.35e-2 ± 1.41e-4	6.33e-2 ± 2.87e-4	6.56e-2 ± 9.63e-4	6.17e-2 ± 3.30e-4	6.36e-2 ± 9.43e-5

   Therefore, we think it is reasonable not to incorporate the NN architecture in the definition of condition number but to focus on the underlying physical challenges.

[1] Funahashi, K. I. (1989). On the approximate realization of continuous mappings by neural networks. Neural networks, 2(3), 183-192.

[2] Sagun, L., Evci, U., Guney, V. U., Dauphin, Y., & Bottou, L. (2017). Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04454.

Q3: Can the condition number be replaced by any scalar in Theorem 3.6?

No, this statement is incorrect. Without further a priori, the condition number is the smallest constant that makes Eq. (10) (see Theorem 3.5 in the latest manuscript) hold. In the proof (see Appendix A.5), we plug Eq. (51) into Eq. (53) to obtain the final result, where the condition number is the smallest constant that makes Eq. (53) hold due to the definition of the supremum.

Q4: In the case where the solution u is linear, what does the condition number reduce to?

According to Definition 3.1, the definition of the condition number is irrelevant to whether the solution $u=u(\boldsymbol{x})$ is linear with respect to the input coordinate $\boldsymbol{x}$.

Introducing an additional theorem

In response to your valuable feedback, we have expanded our theoretical analysis. We introduce a new theorem in the revised manuscript (now in Theorem 3.6) that investigates the global convergence of PINNs under the infinite-width assumption. This analysis utilizes the neural tangent kernel (NTK) framework to establish a link between the condition number and the rate of global convergence, where the factor of the neural network is also considered. We believe this addition addresses your concerns and enriches the paper significantly. The detailed proof is presented in Appendix A.7. We invite you to review this new theorem and look forward to your thoughts.

Q3: Inadequate analysis of sensitivity to hyperparameters and initialization schemes.

Thank you for highlighting the importance of a thorough analysis in this area. We have conducted additional studies to address your concerns, focusing on the impact of various initialization schemes and hyperparameters. These additional analyses strengthen our confidence in the robustness and reliability of our proposed method. The sensitivity to initialization schemes and hyperparameters is minimal, indicating that our approach is adaptable and stable across different settings. This aspect is critical for the practical application of our method in diverse problem contexts.

Different initialization schemes (3 different problems, 3 random trails each):

Different learning rates (Adam optimizer, $\beta_1=0.9,\beta_2=0.999$; the problem is Poisson2d-MS; 3 random trails):

Different betas $\beta_1, \beta_2$ (Adam optimizer, $\eta=1\times 10^{-3}$; the problem is Poisson2d-MS; 3 random trails):

Different number of hidden neural neurons in each layer (the number of hidden layers is 5; the problem is Poisson2d-MS; 3 random trails):

Different number of hidden layers (the number of hidden neural neurons in each layer is 128; the problem is Poisson2d-MS; 3 random trails):

We hope this additional information addresses your concern and demonstrates the thoroughness of our approach.

Q4: How does the condition number help diagnose and rectify training pathologies in PINNs? Any theoretical analysis or empirical evidence?

In Corollary 3.4 (in the latest manuscript), we demonstrate a theoretical link between the condition number and error control in PINNs, indicating that a lower condition number usually results in higher accuracy. This correlation is empirically supported by our experiments shown in Figure 2(b), where a strong relationship (with $R^2 > 0.9$) between the condition number and the error of PINNs is evident across three practical problems. Furthermore, Theorem 3.5 and Theorem 3.6 establishe that the condition number significantly impacts the convergence of PINNs. This is empirically validated in Figure 2(c), which illustrates that PINNs with lower condition numbers converge more rapidly.

Additionally, our preconditioned PINN, as evidenced in Table 2, not only achieves state-of-the-art results in most tested problems but also demonstrates superior performance, reducing errors by an order of magnitude or being the only method to significantly reduce errors below 100% in 7 problems.

Our findings collectively underscore the critical role of the condition number in both diagnosing and improving the training efficacy and accuracy of PINNs.

Q5: What is the significance of the proposed approach for solving complex partial differential equations (PDEs)?

Our proposed approach significantly enhances the performance of Physics-Informed Neural Networks (PINNs) in solving complex PDEs. The condition number, particularly in complex PDE scenarios, is often very high, which directly impacts the accuracy and convergence efficiency of PINNs. As demonstrated in Table 2 of our paper, conventional PINN methodologies exhibit limitations in complex PDE cases like Poisson2d-MS, Heat2d-LT, and Wave2d-MS.

In stark contrast, our preconditioning-based method not only improves error margins but does so dramatically, often by an order of magnitude. This is a crucial advancement, as we've managed to bring error rates consistently below 100% in scenarios where previous baselines failed. Such results underscore the vital role of our approach in resolving intricate PDE challenges, pointing to a substantial leap forward in the applicability and robustness of PINNs in complex mathematical modeling.

Q6: What are the implications of the proposed approach for computational efficiency and scalability?

Our research demonstrates notable improvements in computational efficiency with the proposed method, as evidenced by Figure 3(a). This enhancement stems from utilizing a discretized physics loss, which circumvents the extensive computational demands typically associated with Autograd in continuous physics loss calculations. Notably, the additional computational overhead introduced by our preconditioning algorithm is minimal, typically under 3 seconds, and therefore does not significantly impact the overall training time.

Regarding scalability, Figure 3(b) illustrates that our method exhibits a more favorable scaling of computation time relative to problem size. This finding suggests that our approach holds substantial promise for application in large-scale problems, potentially overcoming some limitations of traditional methods in this domain.

Our results, therefore, indicate that the proposed approach could be a significant step forward in both the efficiency and applicability of PINNs in solving complex partial differential equations.

Dear reviewer tNfv,

Thank you for your valuable comments on several aspects. Here are our responses to address your concerns, which consist of three parts: Part I (Q1-2), Part II (Q3), and Part III (Q4-6).

Q1: Lack of thorough analysis of computational complexity and scalability of the preconditioning algorithm.

Our paper primarily introduces the use of the condition number as a metric for evaluating training pathologies in PINNs. We hypothesize that the ill-conditioned nature of PDEs is a key factor hindering PINNs' effectiveness. This hypothesis is supported both theoretically and empirically, as detailed in Sections 5.2 and 5.3 of our paper. We demonstrate that, without adequate preconditioning, current PINN methods, including advanced baselines, tend to fail (refer to Table 2 and Figure 3(c)).

We believe that our work, even if we employ a well-developed preconditioner, incomplete LU (ILU) factorization, marks a significant direction for future PINN research. It underscores an often-overlooked aspect that could be pivotal in advancing the field.

And we note that the efficiency and scalability of the ILU are well-established in the existing literature. For an in-depth understanding, we recommend references [1] and [2], which provide comprehensive analyses of ILU's performance in large-scale linear systems.

[1] Mittal, R. C., & Al-Kurdi, A. H. (2003). An efficient method for constructing an ILU preconditioner for solving large sparse nonsymmetric linear systems by the GMRES method. Computers & Mathematics with applications, 45(10-11), 1757-1772.

[2] Ghai, A., Lu, C., & Jiao, X. (2019). A comparison of preconditioned Krylov subspace methods for large‐scale nonsymmetric linear systems. Numerical Linear Algebra with Applications, 26(1), e2215.

Q2: Insufficient comparison with other preconditioning methods in the literature.

Thank you for highlighting the need for a more comprehensive comparison with other preconditioning methods. We appreciate this opportunity to provide additional insights from our recent ablation study, which compares several preconditioning methods over three random trials:

The results indicate that the ILU preconditioning method, which we advocate, demonstrates greater stability and effectiveness in comparison to the Row Balancing and Diagonal methods. This evidence supports our choice of ILU as a superior option for the problems we address.

Dear Reviewer,

As we approach the final stages of the review process, we would like to express our gratitude for your insightful comments and suggestions. In response, we have thoroughly revised our manuscript, incorporating several key changes and extending the scope of our research.

In line with your recommendations, we have undertaken comprehensive ablation studies. These include detailed comparisons with alternative preconditioning algorithms and in-depth analyses of our method's sensitivity to various hyperparameters and initialization techniques. Our goal with these additional studies is to bolster the validity and generalizability of our findings.

Your expertise has been instrumental in guiding these enhancements, and we are grateful for your valuable input. We are eager to hear your thoughts on the revisions and are open to further discussions. Should you have additional suggestions, we are fully prepared to consider and implement them.

We look forward to your feedback and hope that our revised manuscript meets the high standards expected by the conference.

Warm regards,

Authors

Q4: Inadequate random samples.

Thank you for pointing out the concern regarding the sample size in our experiments. We have considered your feedback and re-evaluated all experiments using 10 random trials. To succinctly demonstrate the consistency and reliability of our findings, we compared the outcomes of the 5-trial and 10-trial experiments. Our findings show that the results from the 10-trial evaluations align closely with those from the original 5-trial tests, indicating that our initial conclusions are consistent and reliable. Moreover, the comparison with the state-of-the-art (SOTA) baseline methods remains unchanged, affirming the robustness of our approach.

We appreciate your feedback as it has helped us strengthen the validity of our research, and we are confident that the additional results further establish the reliability of our findings.

Q5: why more of the 16 considered problems were not included in Figure 2(b)?

Thank you for this insightful question. We carefully considered the scope and feasibility of our experiments when deciding which problems to include in Figure 2(b). Due to the extensive computational resources required (19-40 instances per problem, each run in 7 random trials), it was not practical to include all 16 problems in this specific experiment.

To ensure a thorough yet efficient analysis, we selected three representative problems that encompass a broad range of characteristics, such as non-linearity and time dependency, where the running time was also considered. These problems were chosen to effectively demonstrate the relationship between the condition number and the accuracy of PINNs, while also maintaining a balance between comprehensiveness and manageability. We believe this approach offers meaningful insight into our method’s capabilities without overextending resources or diluting the focus of our study.

Q6: A few typos in the paper.

Thank you for pointing out these typographical errors. We appreciate your attention to detail, as it helps enhance the quality of our manuscript. We have carefully revised and corrected these errors in the updated version of our paper.

Dear reviewer MgQM,

Thank you for your valuable feedback on improving our paper. Here are our responses to address your concerns, which consist of two parts: Part I (Q1-3) and Part II (Q4-6).

Q1: Scalability to higher-dimensional PDEs.

We appreciate the concern regarding the scalability of our method to higher-dimensional PDEs. While we recognize the limitation of our current approach to dimensions $d\le 4$, as discussed in Section 5.3, this focus aligns with the practicality of addressing many real-world phenomena, which predominantly exist in three-dimensional space.

Furthermore, as illustrated in Figure 3(b), our method demonstrates promising scalability with respect to mesh size. This suggests potential superiority over traditional methods like FEM for larger problems within the $d\le 4$ space.

Significantly, our method excels in addressing inverse problems [1,2], a critical area where physics-informed neural networks have shown distinct advantages over traditional methods. Our studies on two inverse problems from the benchmark [3], as outlined in the table provided, exhibit not only improvements over state-of-the-art PINN baselines but also notable enhancements in accuracy compared to the adjoint method. This is particularly relevant as numerical methods often struggle with noise sensitivity in inverse problems. And the improved performance underscores the method's potential despite its current dimensional limitations.

Our ongoing research aims to extend the applicability of our method to higher-dimensional PDEs with neural-based preconditioners, and we value the feedback to guide our future work.

[1] Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021). Physics-informed machine learning. Nature Reviews Physics, 3(6), 422-440.

[2] Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378, 686-707.

[3] Hao, Z., Yao, J., Su, C., Su, H., Wang, Z., Lu, F., ... & Zhu, J. (2023). PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs. arXiv preprint arXiv:2306.08827.

Q2: The Poisson example in Figure 2(a) is overly simplistic.

Thank you for highlighting the Poisson example in our paper. It's important to clarify that this example is primarily for illustration and not the central focus of our work. We chose a 1D Poisson problem for its analytical simplicity, which aids in clearly demonstrating the behavior and existence of the condition number. This simplicity allows for an analytical expression of the condition number, facilitating a more straightforward discussion and understanding.

We acknowledge that a more complex problem, like a 2D Poisson equation, might be more representative of real-world challenges. However, such complexities could obscure the fundamental principles we aim to illustrate, especially where an analytical expression of the condition number is not feasible.

Our main contributions, as detailed in the paper, are grounded in the two theorems that link the condition number to the error and convergence in PINNs and the comprehensive experiments across a range of problems. These elements of our work address a spectrum of complexities (refer to Table 1) and are not confined to simple PDEs. They illustrate the applicability and robustness of our approach in diverse real-world scenarios, as also acknowledged in the strengths section of your review.

Q3: In Figure 1(a), why does the iteration axis start at 500?

Thank you for your insightful observation regarding Figure 1(a). Upon re-examination, we realized an inadvertent misalignment in the training histories of the compared methods, leading to the omission of the initial iterations. We appreciate you bringing this to our attention. We have rectified this in the updated version of the manuscript to ensure accurate and fair comparison, and have meticulously rechecked the data to prevent similar oversights. The revised Figure 1(a) now accurately reflects the complete training process from the outset.

Dear Reviewer,

As the review process nears its conclusion, we are reaching out to solicit your thoughts on the revisions made to our manuscript, inspired by your valuable insights. In our pursuit of excellence and scientific rigor, we have not only addressed your initial comments but also extended our research scope.

Specifically, we've conducted in-depth experiments on inverse problems and expanded our analysis over a broader range of random trials. These enhancements aim to more convincingly demonstrate the effectiveness and reliability of our approach. Additionally, we have meticulously corrected the identified typographical errors and rectified the graphical issue in Figure 1(a), as reflected in the updated version of our paper.

Your expertise in this domain has been a guiding light in these improvements, and we deeply appreciate your constructive criticism. We are keen to understand your perspective on these latest modifications. Please rest assured that we remain fully committed to further discussions and are prepared to undertake any additional revisions that you may suggest.

We eagerly anticipate your feedback and hope our efforts align with the high standards of the conference.

Warm regards,

Authors