Reply to Reviewer mBxJ

We sincerely thank the reviewer for their thoughtful comments and constructive feedback. We address each concern below and explain how we will improve the manuscript accordingly.

The Unique Role of LLM in Our Framework

The LLM serves several critical and unique functions in PINNsAgent that would be difficult to achieve with smaller language models. To illustrate this, we conducted additional experiments comparing GPT-3.5 and GPT-4 (see table below). The results show that while both models improve over traditional methods, GPT-4 achieves not only better average performance but also significantly lower variance (±8.47E-02) compared to GPT-3.5 (±2.03E-01), demonstrating more consistent and reliable reasoning capabilities. These unique functions include:

1. Reasoning across physics and deep learning domains: The LLM planner must simultaneously understand PDE characteristics (equation type, dimensionality, boundary conditions) and relate them to appropriate neural architectures. This cross-domain reasoning requires the sophisticated knowledge integration capabilities of large language models.

2. Memory Tree Reasoning Strategy (MTRS) implementation: As detailed in Section 3.4, our MTRS approach requires the LLM to function as a policy model π_pl(a|s) that guides the exploration of the hyperparameter search space. The LLM's probabilistic outputs are used directly in the UCT formula to balance exploration and exploitation.

3. Adaptive hyperparameter generation: The LLM analyzes feedback from previous iterations f^(t-1) and adjusts hyperparameters accordingly. This requires understanding complex relationships between hyperparameters (e.g., how learning rate interacts with optimizer choice) and PDE characteristics.

Pipeline for PDEs Not Already in the Code Bank

For PDEs not in the Code Bank, PINNsAgent operates in the Config Generation mode, as the Code Bank contains base code that can be applied to any unseen PDE with the appropriate configuration. The complete pipeline is:

1. PDE encoding: The system encodes the mathematical and physical properties of the new PDE using the comprehensive set of labels described in Section 3.3.

2. Physics-Guided Knowledge Replay (PGKR): PGKR computes weighted cosine similarity between the encoded target PDE and all PDEs in the database. The top-K most similar PDEs are retrieved along with their best-performing configurations.

3. Configuration generation: The planner uses these retrieved configurations as starting points and generates YAML configuration files for the new PDE.

4. Base code application: The system applies the generated configuration to the appropriate base code from the Code Bank. This base code is designed to be flexible and can handle any PDE when provided with the correct configuration.

5. Memory Tree-guided exploration: Following the MTRS approach described in Section 3.4, the system iteratively refines the hyperparameter configuration.

6. Database update: Successful configurations are added to the database to benefit future queries.

Additional Benchmark Datasets

We appreciate the suggestion to evaluate on additional benchmark datasets. As shown in the table below, we conducted experiments on two representative PDEs from the PDEBench dataset: Reaction-Diffusion 1D and Darcy Flow 2D. PINNsAgent consistently outperforms both Random Search and NAS-PINNs on these additional PDEs, demonstrating its broader applicability beyond the PINNacle benchmark.

These additional experiments provide more comprehensive evidence of PINNsAgent's effectiveness and generalizability across different PDE types and benchmarks.

Reply to Reviewer gzrR

We sincerely thank the reviewer for their thoughtful assessment and insightful questions. We address each point below.

Baseline Details and Computational Costs

We ran all experiments on 8 NVIDIA V100 (32GB) GPUs, providing sufficient computational resources to complete all benchmark PDEs in the PINNacle dataset.

Compared to traditional methods, the additional computational overhead in PINNsAgent comes primarily from LLM inference and PGKR retrieval processes. We also observed that the LLM tends to recommend slightly larger (though still reasonable) network architectures, which marginally increases training time. To address the reviewer's concern about computational costs, we conducted additional experiments comparing all methods (Random Search, Bayesian Search, and PINNsAgent) with 5 iterations each. As shown in the table below, PINNsAgent introduces only modest computational overhead (approximately 8.2% compared to Random Search) while delivering substantially better performance.

PDE-Dependent Hyperparameter Sensitivity

Regarding the reviewer's question about hyperparameter variation across PDEs, we found that PINNs exhibit significant hyperparameter sensitivity compared to neural operator methods, even with identical architectures. We found clear patterns in optimal hyperparameters across different PDE types:

Different PDE types consistently favor certain optimizers. For time-dependent diffusion equations (Heat series: HeatND, Heat2D_Multiscale, Heat2D_VaryingCoef), the LBFGS optimizer significantly outperforms other choices. For example, on HeatND, LBFGS (5.64E-08) outperforms MultiAdam (7.93E-08) by approximately 29%. This advantage persists even with increasing problem dimensionality, suggesting that the diffusion mechanism, rather than dimensionality, drives optimizer selection. For static problems (Poisson class), MultiAdam performs better on complex geometries.

The second most important hyperparameter is the activation function. Problems with smooth solutions (Heat, Poisson) benefit most from tanh and sin functions, while problems with sharp gradient changes (Burgers, NS) perform better with gaussian and swish activations. On NS2D_LidDriven, for instance, gaussian (9.89E-06) outperforms sin (1.27E-05).

Network size only needs to be reasonable; excessively large networks do not significantly improve PINN performance but substantially increase computational burden. We also found that advanced architectures like LAAF and GAAF (Jagtap et al., 2020) do not consistently deliver the best performance--the original PINN architecture often proves more robust across different PDEs. These findings validate our approach's ability to identify nuanced relationships through the PGKR framework.

Additional References

We thank the reviewer for suggesting the additional references. We will incorporate these papers in our discussion of related work.

Reply to Reviewer znat

We appreciate the reviewer's thorough assessment of our paper. Below, we address the key concerns raised:

Methods and Evaluation Criteria

1. Knowledge Transfer Evidence: The reviewer questions whether improvements come from learning genuine transferable knowledge or merely from exhaustive hyperparameter search. As shown in our experimental setup (Section 4.1), while baseline methods (Bayesian optimization and random search) required 10 iterations to reach their best performance, PINNsAgent achieved superior results in just 5 iterations across most PDEs. This 50% reduction in required iterations demonstrates efficient knowledge transfer rather than exhaustive search.

2. PGKR's Effectiveness: Our ablation study in Table 2 provides clear empirical evidence of PGKR's effectiveness. When PGKR is removed ("w/o PGKR"), performance degrades significantly across most PDEs. This direct comparison isolates PGKR's contribution to PINNsAgent's performance, demonstrating that the knowledge encoded and retrieved by PGKR substantially improves PDE solving capabilities.

While our approach builds on existing techniques, its novelty lies in the unique integration and adaptation of these methods specifically for PINNs optimization. Our multi-agent LLM framework represents the first comprehensive system to fully automate PINNs development without expert intervention.

Experimental Designs and Analyses

We appreciate the reviewer's feedback on our experimental design and have conducted additional analyses to address these concerns:

Computational Cost Analysis

Regarding computational cost concerns, we conducted additional analysis across all 14 benchmark PDEs, with results shown in the table. The total computation time for PINNsAgent is only about 8.2% higher than random search and 4.1% higher than Bayesian optimization. This additional cost primarily comes from LLM inference and PGKR retrieval processes.

Knowledge Transfer Verification

To demonstrate broader applicability, we also evaluated PINNsAgent on the PDEBench dataset. As shown in the table below, PINNsAgent consistently outperforms both baseline methods on these additional PDEs.

Relation To Broader Scientific Literature We appreciate the reviewer's insights. Our focus on PINNs is deliberate as these models are particularly sensitive to hyperparameter choices - far more than standard neural networks or even neural operators. Small configuration changes in PINNs can lead to order-of-magnitude differences in accuracy, making automated tuning especially valuable for this domain, an observation also confirmed by Wang et al. 2024 in their NAS-PINN work.

• [1] Wang, Yifan, and Linlin Zhong. "NAS-PINN: Neural architecture search-guided physics-informed neural network for solving PDEs." Journal of Computational Physics 496 (2024): 112603.

Essential References Not Discussed

We agree with the reviewer's assessment that our paper's primary focus is on LLM-enabled AutoML for hyperparameter tuning rather than PINNs methodology itself. As noted, we have already included the essential references related to this focus in Section 2.3. We will enhance the literature review by incorporating additional relevant references on hyperparameter tuning approaches to provide a more comprehensive context for our work.

Other issues

1. Definition of L_BC: We agree that L_BC should be formally defined in the Preliminaries section. We will add the boundary condition loss definition alongside the other loss components to ensure completeness.

$L_{BC} = \frac{1}{N_{BC}} \sum_{i=1}^{N_{BC}} \left| u_{\theta}(\mathbf{x}_i^{BC}) - u_{BC}(\mathbf{x}_i^{BC}) \right|^2$

2. Figure reference: Thank you for catching this error. We will correct "Figure ??" to "Figure 2" in line 310.

3. Scope and PINNs architecture: Our framework is designed to be flexible regarding model architecture and training strategies. As mentioned in Section 4.2, our configuration files allow users to specify various PINN variants and training techniques. This includes addressing known issues like spectral bias (Krishnapriyan et al., NeurIPS 2021) through techniques such as curriculum learning, adaptive weighting, and alternative network architectures. The LLM agents can select and configure these options based on the specific PDE characteristics. We will clarify this flexibility more explicitly in the revised manuscript to address this concern.

Reply to Reviewer Yfa3

We sincerely thank the reviewer for their detailed assessment of our paper. We address each concern below:

Novelty and Technical Depth

Our multi-agent LLM framework is the first comprehensive system to fully automate PINNs development without expert intervention. In our pipeline, the Physics-Guided Knowledge Replay mechanism introduces a novel physics-informed approach to knowledge transfer across different PDEs, while our Memory Tree Reasoning Strategy offers a structured exploration method that captures the hierarchical dependencies unique to PINNs design.

LLM Output Reliability and Ablation Studies

Regarding the reliability of LLM-generated hyperparameters, our extensive experiments in Tables 2 and 3 demonstrate that our approach significantly outperforms traditional methods like Random Search and Bayesian Optimization across diverse PDE types. This empirical evidence confirms that our LLM-based reasoning framework adds substantial value to the hyperparameter optimization process for PINNs.

To address the reviewer's concern about different LLM performance, we conducted additional experiments comparing GPT-3.5 and GPT-4. The results show that while both models improve over traditional methods, GPT-4 achieves not only better average performance but also significantly lower variance (±8.47E-02) compared to GPT-3.5 (±2.03E-01), demonstrating more consistent and reliable reasoning capabilities.

Performance comparison with different LLMs (Average MSE across 12 PDEs)

Handling Rare PDE Configurations

For rare PDE configurations, PINNsAgent leverages both the PGKR mechanism and the LLM's reasoning abilities. The PINNacle benchmark used in our evaluation includes a diverse range of PDEs with varying characteristics, including some with complex boundary conditions and multi-scale phenomena. Our consistent performance already across this diverse set demonstrates the framework's robustness to different PDE types.

When encountering a completely new PDE type, the PGKR component retrieves the most similar (though not identical) PDEs from the database, and the LLM uses these as starting points for reasoning about appropriate hyperparameters. The iterative refinement process through MTRS then allows the system to adapt these initial configurations to the specific requirements of the new PDE.

MTRS Implementation Details

We appreciate the request for clarification regarding Equation 7 and the MTRS implementation. In our framework:

1. The initial tree is constructed based on the LLM's knowledge and the most similar PDEs retrieved by PGKR. For each state $s$ (representing a partial hyperparameter configuration), the LLM generates a probability distribution over possible actions $a$ (hyperparameter choices).
2. The policy model $\pi_{pl}(a|s)$ is implemented as the LLM itself. It uses both its pre-trained knowledge and the retrieved similar PDE configurations to generate probabilities for different hyperparameter choices. These probabilities are then used in the UCT formula (Equation 7) to balance exploration and exploitation.
3. The tree is expanded using the standard MCTS process: selection, expansion, simulation, and backpropagation. The key difference is that our expansion and simulation steps are guided by the LLM's output probabilities rather than random sampling or a separately trained policy network.
4. The policy model does not require separate updating since the LLM adapts its recommendations based on the feedback from previous iterations, effectively implementing an adaptive policy.

Other Issues

Thank you for pointing out the typo in line 310. We will correct "Figure ??" to "Figure 2" in the revised manuscript.

We believe these clarifications address the reviewer's concerns and highlight the technical novelty and depth of our approach. The empirical results in Tables 2 and 3, along with the additional LLM comparison in Table 4, provide strong evidence of the effectiveness of our framework.