PhysPDE: Rethinking PDE Discovery and a Physical Hypothesis Selection Benchmark

We sincerely thank the reviewer for the insightful and encouraging feedback. We are delighted that the reviewer finds our approach of leveraging existing physical theories and hypotheses to benchmark PDE learning both interesting and useful. Below, we address the reviewer’s comment in detail.

W1. How your decisions (such as is_compressible, is_newtonian) directly affect the PDE system?

A1. We have updated the Figure 3 (link) to include more details. We also added an introduction to fluid mechanics in the Appendix. C, at Page. 15 of the draft (link), including the conservation laws, compressible flow, and Newtonian fluids, and how they together determine the PDE system. Briefly speaking, 'compressible' means that fluid density is not constant, and 'Newtonian' means that dynamic viscosity is constant. For a more systematic introduction, please refer to the fluid mechanics textbook e.g. [1].

Reference:

[1] White, Frank M., and Henry Xue. Fluid mechanics. Vol. 3. New York: McGraw-hill, 2003.

Thank you for the response, as for your additional questions, we clarify as follows:

1. How to derive PDE terms given a hypothesis?

We start from the full PDE system of the unchangeable physics theorem (e.g. conservation laws in fluid mechanics). On top of that, the derivation is done by plugging the expression of the selected hypothesis into the PDE of conservation laws. All the symbolic calculations are done automatically by a computer algebra system (CAS) library.

For example, the mass conservation is $\frac{\partial \rho}{\partial t}+\nabla \cdot(\rho \mathbf{u})=0$. If one applies the compressible hypothesis, it does not change, but if one applies the incompressible hypothesis, it becomes $\nabla \cdot \mathbf{u} = 0$.

2. How do you constrain the model to only search in equations that are is_compressible=yes?

In the hypothesis search (outer-level optimization), the algorithm can only modify the hypothesis, and can not modify the equation directly. In addition, the hypothesis search is a tree-based algorithm, thus once a compressible hypothesis is selected, then all descendant searches are compressible.

In the new equation discovery (inner-level optimization), the algorithm can modify the equation directly but is confined only to the specific part of the equation allowed by the specific hypothesis.

For example, in our S2 experiment, the new equation discovery is allowed to discover a new expression of viscosity $\mu$ (based on the non-newtonian hypothesis), but is not allowed to modify the compressible hypothesis and its corresponding expression about $\rho$. In our current setting, no new expressions are related to compressibility. Thus the inner-level search will not change compressibility.

Thank you once again for your support and contributions to this review process. If there are any remaining questions or clarifications needed from us, we would be more than happy to assist promptly.

We would like to express our sincere gratitude to the reviewer for thoroughly evaluating our paper and providing constructive feedback. We are pleased that the reviewer recognizes the novelty of our proposed ML4Sci task and the significance of new datasets. Below, we address the reviewer's comments and provide detailed clarifications and responses.

Q1: the release of the dataset and code

A1. In the original submission, we have provided the Fluid dataset link in the supplementary material. Here, we release both the dataset (link) and the code (link) as required. We genuinely hope that the source code can help you better understand our work.

Q2. justification of the use of LLMs in baseline

A2. Selecting physics hypothesis is a brand new ML4Sci task without suitable baseline method. Aside from recovering the symbolic representation, this task further requires the identification of physics properties corresponding to the data. Existing PDE discovery algorithms could only yield symbolic equations and fail to identify physical properties.

To bridge this gap, we take a step back and seek an alternative model that can process both the equation (in math symbols) and hypothesis (in natural language), and can take in hypothesis forest prior knowledge as well. In our experiment, LLMs satisfy these conditions, as they manage to partially select the physics hypothesis given ground truth PDE and the hypothesis forest. Therefore, the pipeline of a classical PDE discovery method and an LLM is a suitable baseline for our new task. We chose GPT-4o because it has the strongest performance in our pre-experiment, among all LLMs accessible to us.

Claim of contribution and significance

Our major contribution is a new ML4Sci task on interpreting scientific data by physics hypothesis. Compared to previous math-only PDE discovery datasets, our PDE hypothesis dataset is additionally labeled with the PDE's physical properties such as compressibility. In addition, a novel algorithm framework is developed for this new task. This algorithm can identify both existing hypotheses and unknown expressions, potentially providing physics insights for scientists and enhancing the automation of physics discovery. Both the datasets and algorithms are publicly available.

Reviewers have recognized our work as "proposing a novel task"(Xb5C), "addressing the crucial challenge of physical interpretability"(QbPL), "interesting and useful"(A96C) and "well motivated"(QC1C).

As we near the end of the discussion period, we would like to thank the reviewer again for the comments and provide a gentle reminder that we have posted a response to your comments. May we please check if our responses have addressed your concerns and improved your evaluation of our paper?

As the discussion period comes to a close, we would like to once again express our sincere thanks for your valuable feedback. We would also like to kindly remind you that we have submitted a response to your comments, updating the data and code links as you requested. Could you please confirm if our responses have addressed your concerns and if they have had any impact on your assessment of our paper?

We are sorry for the misleading 'restricted' mark. The webpage for the data and code is indeed marked as 'restricted,' but it is available by accessing it through the link we provided. You just need to click 'download' to retrieve the files, as shown in Fig1, Fig2. We tested it on multiple computers, operator systems, and web browsers, and the download worked each time. Additionally, in case you are still unable to download it, we have uploaded the code and the training set of the S1 dataset in the supplementary materials. Due to the space limitations for supplementary materials, we could not upload the entire dataset.

We would like to kindly ask if you were able to successfully download the dataset link we provided. If you still encounter any issues accessing it, please do not hesitate to inform us, and we will ensure the link is fully functional.

Additionally, we would appreciate your insights on whether the supplementary materials we submitted have been helpful in evaluating our work. Your feedback is invaluable to us as we strive to improve the quality of our research.

Thank you again for your time and effort in reviewing our manuscript.

I can access the dataset and code.

As the discussion period is coming to a close (only 2 days remaining), we would like to sincerely thank you again for your valuable feedback. We have posted a response to your data availability comments and would appreciate it if you could kindly confirm whether our clarifications have addressed your concerns and helped improve your evaluation of our paper.

As the discussion period is drawing to a close (with less than 1 day remaining), we would like to sincerely thank you once again for your valuable feedback. We have provided a detailed response to your comments regarding data availability, and we would greatly appreciate it if you could kindly confirm whether our clarifications have addressed your concerns and contributed to a better evaluation of our paper. Thank you again for your time and consideration.

As the discussion period is nearing its end (with less than 2 hours remaining), we would like to express our gratitude once again for your feedback. We have provided a detailed response to your comments regarding data availability, and we would be appreciative if you could confirm whether our clarifications have addressed your concerns.

We would like to sincerely thank the reviewers for deciding to increase the score of our manuscript. We are very glad that we were able to address your concerns and appreciate your feedback, which has contributed to the improvement of our work.

We sincerely thank the reviewer for detailed and thoughtful feedback on our work. We are pleased that the reviewer acknowledges our contributions, including the introduction of the new PDE interpretation task and the bi-level optimization algorithm. Below, we carefully address the reviewer's comments point by point.

Q1. Comparison with interpretability-boosted PDE discovery

A1. Existing PDE discovery algorithms, only extract a symbolic expression from data. Certain frameworks, including Champion et al. (2019) and Chen et al. (2021), incorporate "physics" (more precisely, math) priors to boost the accuracy of expression. However, our dataset enables the algorithm to interpret the observation by physics hypothesis. For example, compressibility is an essential property of fluid (see textbook[1]), and one might expect the algorithm to tell whether the data is from compressible or incompressible fluid, instead of finding expressions under a given compressible fluid prior. Hence our method is more flexible and useful to real-world scientific research.

Q2. How to handle high-dimensional cases?

A2. High-dimensional problems primarily result in higher computational overhead. Our main contribution lies in the new task and the new dataset. While PhysPDE serves as a baseline model, it has not been specifically optimized for computational time (we provide some possible speedup techniques below). It can handle the presented datasets (search spaces of size 3000+ and 2D PDEs) within approximately 4 CPU hours which we think is affordable for many practical use.

For higher-dimensional problems, it calls for modifications to the corresponding modules within the PhysPDE framework. For instance, if it is tailored for solving high-dimensional PDEs, the current finite difference algorithm could be replaced with alternatives such as PINN or DeepONet. Similarly, if solving combinations of more hypotheses (i.e., large-scale integer programming problems), deep learning-based methods (e.g. reference [2]) would be a promising alternative. We have to point out that solving the high-dimensional PDEs and large-scale integer programming are long-standing problems in the AI and math community, which is slightly beyond the scope of this benchmark paper.

Q3. How robust is PhysPDE to non-Gaussian noise

A3. We tested on multiple non-Gaussian noises to the S1 dataset and the performance of the top-1 decisions are shown in the table below. One can observe that PhysPDE is generally robust to various noises.

Table A3. PhysPDE S1 performance under various types of noise.

Q4. How the hypothesis decision forest is constructed? Any automation or semi-supervised methods?

A4. The current method for hypothesis design is manual, extracting hypotheses from the table of contents of textbooks and the selection boxes of multi-physics solvers (e.g., COMSOL). We have attempted to use LLMs to automatically design hypotheses, but we found that the results contained logical and mathematical errors. One direction for future work might be to apply knowledge graph extraction to automatically extract hypotheses from textbooks and papers.

Q5. What if multiple plausible hypotheses exist? Probabilistic ranking or ensemble approach?

A5. There are two mechanisms to handle the case of multiple plausible hypotheses:

1. Occam's Razor: simpler hypotheses are prioritized. This is implemented as a soft selection by adding complexity penalty terms (Eq. 3).
2. Ensemble Learning: The data is randomly divided into groups, and for each group, only the top 3 hypothesis tree search results are retained. The final results are selected based on the frequency of occurrence of the outcomes. The results reported in the paper are the ones with the highest frequency.

Q6. How to evaluate interpretability and physical alignment beyond standard metrics?

A6. Standard evaluation metrics find matching between mathematical expressions, with precision and recall defined for the set of terms in the expression. In comparison, we consider the matching degree of hypothesis selection, and the precision and recall in our paper are defined for the set of hypotheses. Based on our tree structure, one can also extend the evaluation metrics, such as using the shortest path length between decision nodes or the height of the lowest common ancestor as additional metrics.

Reference:

[1] White, Frank M., and Henry Xue. Fluid mechanics

[2] Nair, Vinod, et al. Solving mixed integer programs using neural networks.

We sincerely thank the reviewer for the thoughtful feedback and constructive comments. We are delighted that the reviewer recognizes the novelty of our work in addressing the new task of explaining observed data using physics domain knowledge and theories in this context. Below, we address the specific questions and suggestions raised by the reviewer in detail.

W1. need more explanation in method, figure 2 and figure 3.

A0. We have modified the draft based on your suggestions. The modification in the main text is marked in blue in the updated draft (link). A detailed explanation of the algorithm is in Appendix E&F. We also updated a new version of Figure 3 (link), with an example of constructing a PDE system from hypotheses.

Q1. is this work limited to discovering only known theories

A1. Our work can discover new expressions on top of the known theories, if new expressions (empirical formulas) could be regarded as a kind of new theory. The example you mentioned, discovering unknown non-Newtonian fluids given knowledge of Newtonian fluid, is exactly the S2 experiment in the paper (see the second column of Table 9).

Q2. benefits compared to previous empirical law discovery algorithms

A2. The benefits are at least twofold: interpretability and empirical performance.

1. previous empirical law discovery algorithms like BACON only produce pure math expressions like y=f(x), without any physical interpretation such as Newton's laws of motion, Fourier's law of heat conduction, etc. Our work is based on the physical hypothesis search and produces a physical interpretation of data, which is critical for real-world physics research.

2. the incorporation of physics laws boosts the model's performance. As shown in Table 5-8, our model outperforms existing math-only models with a clear margin in both accuracy and simplicity.

Q3. future use of the dataset

A3. The dataset can be used in both AI and physics communities.

• AI. The newly defined physical hypothesis discovery task can encourage the design of models with more physical interpretability. The two datasets we provide serve as a benchmark for these new models. For instance, our current experiments demonstrate that GPT-4o (Table 10) performs poorly on the physical hypothesis selection task. A potential research direction is to explore how to improve existing large language models to handle such tasks and datasets, thereby advancing the development of AI-powered physics research assistants.

• Physics. In the field of physics, experts can leverage their domain knowledge alongside our existing frameworks and algorithms to conduct cutting-edge scientific exploration. For example, by incorporating radiation heat transfer theory into fluid dynamics, researchers can investigate the mechanisms of nuclear fusion reactions (Section 5). The model is capable of automatically selecting the most plausible physical hypotheses based on experimental or simulation data, offering both inspiration and evidence for scientific discovery.

Q4. is there a metric that can measure the interpretability, or do we need domain expert opinion?

A4. We do not rely on domain expert opinion on the measurement. The evaluation metrics related to interpretability in this benchmark are objective and quantitative:

1. Match of hypothesis with the truth, evaluated by precision and recall between the selected and true hypothesis. In the future, one can also extend the evaluation metrics based on our tree structure, such as the shortest path length between decision nodes or the height of the lowest common ancestor.

2. Complexity of hypothesis, evaluated by the size of the hypothesis set. The hypothesis should be as simple as possible, according to the Occam's razor.