MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Data

[Q10] Although the extrapolation conducted in this task was on a smooth domain, we found it meaningful as our model outperformed other baselines and also showed superior performance compared to neural operator baselines in the additional 2D CDR experiment. More challenging experiments, such as boundary layer problems, are important topics in their own right and will be addressed in future work.

Additionally, to evaluate our model's performance on a PDE exhibiting singular behavior in its solution profile, we conduct an extrapolation task experiment on the Burgers equation with shock formation. In this experiment, we compare the performance of our model with FNO and DeepONet. Our model outperforms DeepONet but falls short of FNO. This indicates that our model's cross-attention structure may be insufficient to infer unexpected patterns.

To address this limitation, we adopt the approach suggested by Reviewer FgQz, incorporating the PINN loss into our model during training. As a result, the enhanced model outperforms the original model across all metrics and surpasses FNO in two metrics. This demonstrates the potential of PINN loss as a promising direction for improving our model in future work. Detailed information about this experiment has been added in Appendix K of the revised paper.

[Q11] Before dealing with more challenging problems, it is essential to verify that our model can accurately predict the linear combination of various reaction terms. To this end, we expand our experiments to explore nonzero values for both convection and diffusion terms. In other words, we conduct additional experiments on 1D CDR with diverse reaction terms using fine-tuning. We identify that while our model performs well with only pre-training, fine-tuning can further improve its performance when higher accuracy is desired. Detailed information about this experiment has been added in Appendix L.2 of the revised paper.

[Q12] Apologies for the confusion. What we intended was the generalization within the CDR framework to unseen parameters, rather than to completely different types of unseen PDEs. We have fixed this sentence in the revised paper based on your feedback.

[Q13] Apologies for not including experimental details in the main text of the paper. As mentioned in the general response, we have added details of the experiments in Appendix N of the revised paper. During the training phase, we sampled 800 collocation points, 256 initial condition points, and 200 boundary condition points to train our model. In the test phase, we used 200 collocation points as observed data to infer solutions for 1,000 non-overlapping points where the solution is unknown.

[Q14] You are absolutely correct that fine-tuning or few-shot learning can further improve model performance. However, in this paper, especially in Section 4.3, we aimed to emphasize the model’s capability for in-context learning when trained with a linear combination of different reaction terms. This approach is a novel yet essential attempt to expand the parameter space and demonstrate the model’s potential to perform inference effectively even in a zero-shot setting. We wanted to highlight this possibility, which we believe is a valuable step towards more flexible and generalizable PDE solution inference.

The 6% error mentioned is due to the tested parameter space. In the last three columns of Table 4, the PDE parameters tested were not seen during training, while the first column includes parameters seen in training. Additionally, the range of the tested PDE parameters lies outside the range of the training parameters, making it an extrapolation of the parameter space. Therefore, we believe this error does not directly critique our model’s in-context learning capability. To address this, we have added an additional experiment with fine-tuning as a complement in the revised paper. Detailed information about this experiment has been added in Appendix L.1 of the revised paper.

Thank you for the valuable feedback on our paper. We hope that our responses will address the reviewer’s questions effectively.

[W1] As you mentioned, the referenced papers do utilize low-fidelity data. However, their approaches and objectives differ from those of our model. We have compared these aspects in the table below.

First, we compare in terms of whether high-fidelity data is used during the training phase. Our model can be properly trained without any high-fidelity data during training. On the other hand, the other two models use high-fidelity data during training. Second, we compare in terms of whether information about the governing equation is utilized. In our model, no information about the governing equation is used at any stage. Our model learns solely based on observed data. However, the other two models use the PINN loss during training. Third, we compare in terms of whether the model solves multiple PDEs. Since our model is a foundation model, it is trained with priors generated from multiple PDEs. On the other hand, the other two models are trained with data from only one target PDE. Lastly, we compare in terms of whether the model solves PDEs with unseen parameters. Our model can generalize to PDEs with unseen parameters. However, the other two models are tested only on the trained PDE.

[1] Meng, Xuhui, and George Em Karniadakis. "A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems." Journal of Computational Physics 401 (2020): 109020.

[2] Howard, Amanda A., et al. "Multifidelity deep operator networks for data-driven and physics-informed problems." Journal of Computational Physics 493 (2023): 112462.

[W2] As you mentioned, since our original paper only presented experiments on 1D CDR, it was necessary to explore the broader potential of our model. In response to this necessity, we extend the dimension to conduct experiments on 2D CDR. Moreover, we conduct experiments on the biharmonic equation and Burgers' equation. In these experiments, our model demonstrated relatively strong performance compared to the baselines. Details for the experiments can be found in Appendices I, J, and K of the revised paper. While we have not pre-trained the model on a variety of complex PDEs as suggested, these additional experiments confirm that our model is not limited to working effectively on only 1D CDR.

[W3] We used the CDR equation to simplify the experimental setup and for comparison with existing methods such as Hyper-LR-PINN and P$^2$INN. However, our model did not receive any information about the equation during training. It was trained solely on observed data points sampled from the prior, and this reflects real-world scenarios.

In the 1D CDR experiments, we conducted tests with coefficients $\beta, \nu, \rho \in [1, 20] \cap \mathbb{Z}$ , which resulted in a dictionary containing 8,000 PDEs. For the 2D CDR experiments in Appendix I of the revised paper, we use the coefficients with $\beta_x, \beta_y, \nu_x, \nu_y, \rho \in [1, 3] \cap \mathbb{Z}$, which formed a dictionary of 243 PDEs. In both cases, we believe that we have demonstrated the potential of our model as a scientific foundation model by using sufficiently large dictionaries.

Additionally, we conduct experiments not only on CDR but also on Burgers' equation and the biharmonic equation, proving that our model is not limited to CDR applications. The details of the experiments are in Appendix J and K in the revised paper.

Dear Reviewer,

The authors have provided their rebuttal to your comments/questions. Given that we are not far from the end of author-reviewer discussions, it will be very helpful if you can take a look at their rebuttal and provide any further comments. Even if you do not have further comments, please also confirm that you have read the rebuttal. Thanks!

Best wishes, AC

Thank you for the valuable feedback on our paper. We hope that our responses will address the reviewer’s questions effectively.

[W1] Please refer to the general response. We have added description of our model and a simplified figure illustrating the detailed model structure in Appendix O of the revised paper. We also have revised Figure 2 in the revised paper.

[W2] Please refer to the general response. We have incorporated an experiment involving the biharmonic equation with higher-order derivative terms and the Burgers equation with solution profile that exhibit singular behavior. These additions demonstrate our model's capablility for in-context learning across a diverse class of PDEs.

[Q1] The concept of "zero-shot inference" in our paper refers to inferring solutions for a partial differential equation (PDE) without additional information except the target PDE itself. Unlike the in-context operator learning (ICON) introduced by L. Yang et al. (2023), our model does not rely on "demos" (few-shot examples with various initial/boundary conditions regarding the target operator to learn) during inference. ICON learns using diverse conditions and quantity-of-interest (QoI) pairs, i.e., demos, for a hidden PDE and employs these demos to predict answers for a new test condition, which is a few-shot learning.

In contrast, our model infers solutions to unseen PDEs solely based on the prior knowledge learned from a set of prior PDEs. This prior knowledge encapsulates generalizable patterns from the solution space of PDEs, enabling zero-shot inference without additional demos. The prompts shown in Figure 2, which consist of the information about observed solution points, represent the only input provided for the unseen PDE. Importantly, these prompts are not analogous to few-shot demos as used in ICON. This distinction underscores the zero-shot capability of our approach, which eliminates the need for PDE-specific training or demo-based fine-tuning.

[Q2] In machine learning, the $L_1$ absolute error, $L_2$ absolute error and $L_2$ relative error are commonly used as a base metric. For example, [1-2] used the $L_2$ relative error, [3] employed the $L_1$ absolute error, and [4] utilized the $L_1$ absolute error, MSE, and $L_2$ relative error. Thus we have used the $L_1$ absolute error and $L_2$ relative error. Also, we used $L_2$ relative error for performance comparison with methodologies presented in other papers, as this metric enables a direct comparison. Instead of the $H_1$ norm, we have additionally provided the $L_{\infty}$ relative error in the revised paper.

[1] Kang, Namgyu, et al. "Pixel: Physics-informed cell representations for fast and accurate pde solvers." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 7. 2023.

[2] Lee, Jae Yong, Sung Woong Cho, and Hyung Ju Hwang. "HyperDeepONet: learning operator with complex target function space using the limited resources via hypernetwork." arXiv preprint arXiv:2312.15949 (2023).

[3] Herde, Maximilian, et al. "Poseidon: Efficient Foundation Models for PDEs." arXiv preprint arXiv:2405.19101 (2024).

[4] Yang, Liu, et al. "In-context operator learning with data prompts for differential equation problems." Proceedings of the National Academy of Sciences 120.39 (2023): e2310142120.

Dear Reviewer,

The authors have provided their rebuttal to your comments/questions. Given that we are not far from the end of author-reviewer discussions, it will be very helpful if you can take a look at their rebuttal and provide any further comments. Even if you do not have further comments, please also confirm that you have read the rebuttal. Thanks!

Best wishes, AC

Dear Reviewer WwXL,

We are glad that your questions regarding zero-shot inference have been resolved. We also appreciate your understanding of the metrics we have used. Nevertheless, to address your point about the necessity of the H1 norm, we have attempted to evaluate the error using the H1 norm during the rebuttal period for the 2D CDR interpolation task. For this evaluation, we use the settings that had previously shown the best performance for each model and evaluate on a new test dataset in the form of grid data.
The results have been incorporated into Appendix I.2 and I.4 of the revised paper. The results demonstrate that our model outperforms the baselines with respect to the H1 norm as well.

We are currently working on evaluating other experiments using the H1 norm. We will do our best to present these results before the discussion period concludes.

Dear Reviewer WwXL,

As mentioned in our previous additional response, we have evaluated the $H_1$ norm in another experiment. This experiment was conducted on the time domain extrapolation task with unseen parameters in 1D convection equation. The results are as follows:

Since this experiment was conducted on equations with unseen parameters, the absolute values are slightly higher. However, our model still demonstrates superior performance compared to the baselines. These results will be included in the paper in the future, as the deadline for revising the current paper has already passed.

The discussion period end on December 2(AoE). If you have any additional questions, please let us know before the deadline. We have continuously worked throughout the discussion period to address your concerns and questions.

Best regards,
The Authors

Thank you for the valuable feedback on our paper. We hope that our responses will address the reviewer’s questions effectively.

[W1, W2, Q1, Q2] Please refer to the general response. We have added content about experiments for 2D CDR. Moreover, we have also included the computational cost analysis for 2D CDR, allowing us to observe how the computational cost increases as the dimension grows in our model.

[W3] We apologize that some details are in the Appendices. The reason we prioritized including the current results in the main text was to emphasize the performance of our model in various tasks leveraging in-context learning (ICL). Due to space limitations in the main text, contents related to cost, training algorithms, and failure modes had to be moved to the Appendices.

[Q3] First, thank you for your insightful question. In the extrapolation task experiment on the Burgers' equation, we apply the PINN loss as a physical constraint to our model. As a result, it demonstrates better performance compared to the original version of our model. This highlights the potential of utilizing the PINN loss as a means of improving our model’s ability to predict unexpected patterns, addressing one of the limitations of its cross-attention structure. Additional experiments supporting this approach are detailed in Appendix K. Thanks to your valuable question, we have identified a promising approach to further enhance our model in the future. Thank you again.

[Q4] The convection-diffusion-reaction (CDR) equations studied in our work are commonly used across various physical and chemical domains. For example, they are widely utilized in environmental engineering, such as modeling pollutant dispersion in air or water and studying mixing processes in rivers [1]. They also serve as fundamental models for heat and mass transfer in fluid flows [2]. Furthermore, the reaction terms considered in our study are not arbitrary. Each reaction term is commonly used in specific domains. For instance, the Fisher reaction term describes the spatial spread of advantageous genes through populations, forming the foundation for modeling phenomena like biological invasions and genetic propagation [3]. The Allen-Cahn reaction term is a fundamental model in materials science for simulating phase separation in metals or fluids [4].

Beyond the scenarios directly covered in the paper, our methodology has the potential for further expansions. For the sparse sensor data completion, our method can be applied to reconstruct missing data from sparse sensor measurements. This frequently happens for i) environmental monitoring, where filling spatial or temporal gaps in pollutant or weather data is crucial, ii) scientific simulations, where generating complete datasets from partial observations can accelerate simulations and/or improve experimental outcomes. These real-world examples and extensions demonstrate the broad applicability of our approach and provide a foundation for exploring new tasks in diverse scientific and engineering fields.

[1] Jennifer G. Duan, S.K. Nanda, "Two-dimensional depth-averaged model simulation of suspended sediment concentration distribution in a groyne field", Journal of Hydrology,Volume 327, Issues 3–4 (2006).

[2] POSTELNICU, Adrian. "Influence of chemical reaction on heat and mass transfer by natural convection from vertical surfaces in porous media considering Soret and Dufour effects". Heat and Mass transfer, 43.6: 595-602 (2007).

[3] Fisher, Ronald Aylmer. "The wave of advance of advantageous genes." Annals of eugenics 7.4: 355-369 (1937).

[4] Tourret, Damien, Hong Liu, and Javier LLorca. "Phase-field modeling of microstructure evolution: Recent applications, perspectives and challenges." Progress in Materials Science 123: 100810 (2022).

Thanks for your valuable feedback. We have added experiments related to the theorem, providing not only a mathematical proof but also empirical evidence.

[W1] As mentioned in the general response, we have added the description of our model and a simplified figure in Appendix O of the revised paper. We have also revised Figure 2 in the revised paper.

[W2] As we mentioned in the introduction, various models are already categorized as Scientific Foundation Models (SFMs). However, each of these models covers only specific tasks in the realm of science. For example, [1, 2] focus solely on tasks related to the medical domain, and [3] is dedicated to audio-related tasks. Similarly, solving PDEs is a subset of science, and it is a highly studied area. Consequently, models like [4, 5] are referred to as SFMs for solving PDEs. Therefore, as our model also belongs to this category of foundation models for solving PDEs, we described it as an SFM. However, we will make it clear that our model is a scientific founcation model for solving PDEs.

[1] Xie, Qianqian, et al. "Me llama: Foundation large language models for medical applications." arXiv preprint arXiv:2402.12749 (2024).

[2] Moor, Michael, et al. "Foundation models for generalist medical artificial intelligence." Nature 616.7956 (2023): 259-265.

[3] Yang, Dongchao, et al. "Uniaudio: An audio foundation model toward universal audio generation." arXiv preprint arXiv:2310.00704 (2023).

[4] Yang, Liu, et al. "In-context operator learning with data prompts for differential equation problems." Proceedings of the National Academy of Sciences 120.39 (2023): e2310142120.

[5] Herde, Maximilian, et al. "Poseidon: Efficient Foundation Models for PDEs." arXiv preprint arXiv:2405.19101 (2024).

[Q1] As mentioned in the general response, we have added an experiment on the model's sensitivity to the number of the given data $\widetilde{D}$ during testing in Appendix M of the revised paper. This result demonstrates the sensitivity of the neural network's posterior distribution approximation to the data size ($D_n$). As the data size increases, the network's approximation to the posterior distribution becomes quickly accurate, converging to the expected value under the prior distribution. This sensitivity to the data reflects the consistency and robustness of the Bayesian inference process.

Our model leverages this observation by performing Bayesian inference that incorporates prior data, allowing it to infer spatiotemporal points that align with an appropriate partial differential equation (PDE) solution under given conditions. Experimental results in Appendix M confirm this behavior, showing how, as ($D_n$) increases, the network’s solution converges to the true underlying solution.

Dear Reviewer,

The authors have provided their rebuttal to your comments/questions. Given that we are not far from the end of author-reviewer discussions, it will be very helpful if you can take a look at their rebuttal and provide any further comments. Even if you do not have further comments, please also confirm that you have read the rebuttal. Thanks!

Best wishes, AC