PICL: Incorporating Coarse-Grained Data and Physics Information for Superior Physical Systems Modeling

Thank you for your valuable comments. We have conducted several additional experiments and comprehensively revised our paper to address your valuable questions and suggestions. Below are our detailed responses to each of your questions and concerns:

Question about what does the prediction task mean and solution for incomplete information.

A: Thank you for your question.

• Your inference 'a' is correct. In our paper, "predictive ability" refers to the capacity to predict the subsequent coarse-grained observations from the current coarse-grained observation. In light of your question, we have now included a clear definition and explanation of "predictive ability" in both the Abstract and the concluding paragraph of the Introduction.
• For your mentioned incomplete information, we use coarse-grained observations from previous steps to estimate the current fine-grained state inspired by the finite difference method, to compensate incomplete information caused by coarse-grained observed data. We added the description discussion of our method in Section 4.2.1.
• Furthermore, for the impact of historical data length, we already did the ablation study and analyzed it in the third paragraph of Section 5.5.

Question about why the transition module is required.

A: Thank you for your question. We also considered this aspect at the inception of our method's design, primarily for three reasons:

• Firstly, numerous related studies, such as FNO [1], MAgNet [2], DINo [3], NeuralStagger [4], and others, focus on approximating numerical solvers using neural networks. Consequently, we decided to follow the ideas of these works, opting for neural network approximations rather than directly implementing numerical solvers.
• Secondly, in response to your question, we have conducted an additional experiment to compare the computational costs of the neural network in the transition module versus a numerical solver, which demonstrates the value of the transition module. This comparative study is detailed in the Appendix G. The findings are as follows: The time taken for a single computation using a numerical solver is more than five times that of a step inference with a neural network. Furthermore, as the resolution increases, the computational cost of using a neural network does not significantly change, as only the input and output layers are affected, with small changes in the hidden layers. These results indicate that, in terms of computational cost, whether in training or inference, employing a neural network in the transition module offers considerable advantages.
• Thirdly, employing a differentiable neural network is important for the future expansion of our proposed method. This approach may be beneficial for addressing control problems and inverse problems in subsequent research.

[1] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A. and Anandkumar, A., 2020. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895.

[2] Boussif, O., Bengio, Y., Benabbou, L. and Assouline, D., 2022. MAgnet: Mesh agnostic neural PDE solver. Advances in Neural Information Processing Systems, 35, pp.31972-31985.

[3] Yin, Y., Kirchmeyer, M., Franceschi, J.Y., Rakotomamonjy, A. and Gallinari, P., 2022. Continuous pde dynamics forecasting with implicit neural representations. arXiv preprint arXiv:2209.14855.

[4] Huang, X., Shi, W., Meng, Q., Wang, Y., Gao, X., Zhang, J. and Liu, T.Y., 2023. NeuralStagger: accelerating physics-constrained neural PDE solver with spatial-temporal decomposition. arXiv preprint arXiv:2302.10255.

Suggestion in formulas of loss terms and metrics.

A: Thank you for your insightful suggestions. We have made revisions accordingly.

• In Section 4.2.2, we have added more clearly definition each component of the loss function, including data loss and physics loss, using precise mathematical formulations.
• Furthermore, in the first paragraph of Section 5, we have introduced a formal definition for the reconstruction error $\epsilon$ using a mathematical equation, replacing the previous textual description.

Question about handling non-one-to-one mapping.

A: Thank you for your comments. I have clarified a few points regarding the non one-to-one mapping issue:

• Due to the partially observed nature of the problem, we do not assume a one-to-one mapping between the current state $u_t$ and the next state $u_{t+1}$.
• Instead, as described in Section 4.2.1, we use coarse-grained observations from previous steps to estimate the current fine-grained state. There are two interpretations of this: First, similar to PDEs, we can use the temporal derivative estimated from historical data to approximate the spatial derivative and reconstruct the current fine-grained state. Second, we assume a one-to-one mapping exists between a sequence of historical coarse-grained observations and the current fine-grained state.
• We performed an ablation study (third paragraph, Section 5.5) analyzing the impact of the length of historical data used. This provides some empirical analysis of the mapping between observations and states.
• As you pointed out, uncertainty-aware models like Bayesian neural networks and variational autoencoders are promising approaches for handling partial observations. I have added a discussion of these methods in Section 4.2.1.

Question about multi-step performance.

A: Thank you for your question. Please note we have already show the multi-step prediction tests and compared them with baselines in our main paper. Please refer to Sections 5.2, 5.3, and 5.4, with results illustrated in Figure 2. To avoid confusion, we have revised the first paragraph of Section 5 by following your suggestion.

Suggestion about sentence expression.

A: Thank you for your valuable suggestion. We have revised Section 4.2.1 to more accurately describe the transition module's role in the prediction phase. To be more clear, this sentence means the transition module can achieve the forecasting function. We design a neural network that inputs the current fine-grained state $\hat{u}\_{t}$ to forecast the subsequent fine-grained state $\hat{u}\_{t+1}$.

Thank you again for your comments and suggestions to significantly improve the quality of our paper. We have read your comments carefully and understood your concerns. To address your concerns, we have revised several sections of our paper and conducted the additional experiments. Do you have any other concerns or suggestions?

Thanks for your reply and suggestions to improve our work. We have conducted several additional experiments and comprehensively revised our paper to address your valuable questions and suggestions. Below are our detailed responses to each of your questions and concerns:

Concern about limitations.

A: Thanks for your question.

• For the limitation of our proposed method: In Appendix D of our paper, we discuss the limitations of our work.
• The first one is the compatibility of our proposed framework and the specific neural network architecture. We have conducted addtional experiments and made a detailed discussion in the 11st paragraph of Appendix B.6.
• The second one is how to incropreate the avalable fine-grained data. We have conducted addtional experments and made a detailed discussion in the last paragraph of Appendix B.6. You may also check the following Question and Answer.

Questions about the usage of fine-grained data.

A: Thanks for your question.

• For aspect of how fine-grained data can be incorporated into our framework:
  • Our current work is primarily focused on scenarios where only coarse-grained data is available, aligning with a significant portion of both simulation or real-world applications.
  • We conjecture that if we assume the fine-grained data is available, the predictive performance will be better.
  • To verify the above conjecture, we conducted an additional experiment incorporating fine-grained data. We detailed the experiment and its results in last paragraph of Appendix B.6. In this experiment, fine-grained data was used as the label for the encoding module, aiming to train the module to reconstruct a more accurate learnable fine-grained state. Our findings show that both the baseline FNO* (without physics loss) and our proposed PICL benefit significantly from the inclusion of fine-grained data, with improvements of over 38% and 12% respectively.
  • Moreover, these results also highlight an important aspect of our framework: the ability to use the known physics information as a substitute for fine-grained data in training the encoding module, especially in scenarios where such data is unavailable. The smaller improvement observed in PICL compared to the FNO*, when fine-grained data is available, suggests that PICL can effectively compensate for the absence of fine-grained data to learn the state. Such discussion is also placed in Appendix B.6.

Question about why some results of baselines outperform our method.

A: Thank you for pointing this issue.

• For $\epsilon$ of PINO* outperforms PICL in Table 1, based on our experimental log data, we realized that there is a typo. The value should be 3.18E-1 instead of 3.18E-3. We have re-run the experiment multiple times, confirming that the magnitude is indeed in the E-1 range. We have fixed it in the table and extend our gratitude for your attention to this clerical error in our paper.
• To avoid you from being confused. In Table 1, the PIDL's $\epsilon$ for LSWE performs better than PICL. We have explained it in Section 5.3, before. Even though the $\epsilon$ value in PIDL is slightly lower, this approach concentrates on training using only the physics loss, overlooking the importance of data constraints in predictions. Consequently, PICL has been developed to trade off between data and physics constraints, thereby improving the model's predictive ability. We consider this current observation to be normal, as our method aims to enhance the model's predictive ability using physical information in coarse-grained data, which is reflected in the $\mathcal{L}_{d}$ in our paper. The $\epsilon$ is only a metric to show the accuracy of the reconstructed fine-grained state, a step in our process, but not the end goal of our work.

Suggestion about the lengthy introduction of methodology.

A: Thank you for your valuable suggestion, which has significantly contributed to the clarity and readability of our paper. We have revised Section 4 accordingly :

• Firstly, we have succinctly introduced our concept and methodology in Section 4.1, using clear and easily understandable language to provide an overall view. This approach ensures that readers can quickly grasp the essence of our work.
• Secondly, for those readers interested in the technical specifics, we delve deeper into the details of our method in the subsequent Section 4.2.

Thanks for your recognition of our definition of the problem. We have conducted several additional experiments and comprehensively revised our paper to address your valuable questions and suggestions. Below are our detailed responses to each of your questions and concerns:

Question about how physics loss is defined.

A: Thank you for your valuable feedback regarding our experiments.

• According to your question, we have added a more detailed explanation immediately following Eqn. 5 in Section 4.2.2. The enhancement includes an in-depth description of the physics loss and the introduction of the RK4 method are placed in Appendix B.2.
• We first present the process of RK4, which can be considered as a function $F(u_t,u_{t+1})=0$. By using $F$, we can design two physics losses $\mathcal{L}\_{ep}(\theta)=F(u_t(\theta), u_{t+1}(\theta))^2$ for the encoding module and $\mathcal{L}\_{tp}(\omega) = F(u_t, u_{t+1}(\omega))^2$ for the transition module. By minimizing the $\mathcal{L}\_{ep}$ and $\mathcal{L}\_{tp}$, the $u_t$ and $u\_{t+1}$ can be found, which means that the $u\_t$ and $u\_{t+1}$ can satisfy the RK4 time differentiation.

Suggestion of adding baseline.

A: Thank you for your suggestion.

• Because FNO and PINO are common methods that many works about neural operator used as baselines, we previously compared our method with them.
• Thank you for mentioning MAgNet. We have done further literature review and found the work of three state-of-the-art neural operators, including MAgNet [1], DINo [2], and LatentNeuralPDEs [3], and cited them in the Related Work. Considering the limitations of computing resources, we selected the method LatentNeuralPDEs with the best performance as our baseline.
• We have compared our proposed method with the official implementation of LatentNeuralPDEs. The details about LatentNeuralPDEs are described in Appendix C, and the comparative results are presented in Appendix E and Table 9. The findings indicate that the loss ($\mathcal{L}_{d}$) of LatentNeuralPDEs is larger than that of PICL w/o fine-tune and PICL with fine-tune by more than 81% and 91%, respectively.

[1] Boussif, O., Bengio, Y., Benabbou, L. and Assouline, D., 2022. MAgnet: Mesh agnostic neural PDE solver. Advances in Neural Information Processing Systems, 35, pp.31972-31985.

[2] Yin, Y., Kirchmeyer, M., Franceschi, J.Y., Rakotomamonjy, A. and Gallinari, P., 2022. Continuous pde dynamics forecasting with implicit neural representations. arXiv preprint arXiv:2209.14855.

[3] Iakovlev, V., Heinonen, M. and Lähdesmäki, H., 2023, November. Learning Space-Time Continuous Latent Neural PDEs from Partially Observed States. In Thirty-seventh Conference on Neural Information Processing Systems.

Suggestion of more PDEs experiments.

A: Thank for your suggestion. We agree that additional experiments involving PDEs used in FNO are essential. Regarding your question about the 1D case decay flow, you might mean the Burgers flow experiment in FNO, as decay is a characteristic feature of the viscous Burgers equation? If not, please tell me what you mean.

• Based on your suggestion, we have conducted experiments on the Burgers Equation and the Navier-Stokes Equation following the settings in original FNO paper, and compared the results with FNO. The detailed experimental setup and results are presented in the Appendix F of our paper.
• We summarize the results as follows: In the Burgers Equation experiments, the data loss ($\mathcal{L}_{d}$) of PICL with fine-tuning is marginally lower than that of FNO* and FNO. In the Navier-Stokes Equation experiment, which involves a more complex PDE, PICL with fine-tuning demonstrates significant improvement over both FNO* and FNO. These results are detailed in Table 10. This enhancement in performance shows the efficacy of our approach and its potential in handling intricate fluid dynamics problems.

Thank you again for your comments and suggestions to significantly improve the quality of our paper. We have read your comments carefully and understood your concerns. To address your concerns, we have revised several sections of our paper, and conducted numerous experiments on the additional baseline, different PDEs, and zero-shot super-resolution cases. Do you have any other concerns or suggestions.

Thank you for your valuable reply. We have conducted several additional experiments and comprehensively revised our paper to address your valuable questions and suggestions. Below are our detailed responses to each of your questions and concerns:

Question about the problem setting and its importance.

A: Thank you for your question. To make our paper clearer and easier to follow, we have made the following adjustment to the paper:

• For the problem setting, the scientific problem we solved is how to apply physics information on coarse-grained data to improve predictive ability, whether in numerical simulations or real-world applications when we only have coarse-grained data. We revised the Abstract and the 2nd paragraph of the Introduction to make it clear.
• For its importance:
  • In real-world scenarios, only coarse-grained data measured by sensors is available that is the importance of the problem setting.
  • Generally, this framework can be applied to any problem seeking to enhance model predictions using physics information on coarse-grained data, offering insights and potential solutions.

Differences between PICL, FNO*, and impacts of each term in loss.

A: Thank you for your question.

• We have made revisions to Section 5.1, first paragraph, to provide a more detailed description to each baseline for better understanding. Here to clarify: To control variables, PICL and the FNO* baseline are very similar in terms of network structure. Specifically, the primary difference between PICL and FNO* is that FNO* omits the physics loss components from the PICL’s loss function, retaining only the data loss.
• Regarding the impact of each term in loss functions, our paper already discussed this in Section 5.5 Ablation Studies, specifically in the second paragraph. Our experimental findings indicate that including a physics loss in the loss function leads to better results compared to method such as FNO* without it. The model performs optimally when the physics loss coefficient, $\gamma$ = 1E-1. This is because a very low weight for physics loss makes it ineffective in the optimization process, while a very high weight causes the model to overly prioritize satisfying physical constraints at the expense of predictive accuracy. We appreciate you highlighting this issue, and we have made revisions to this section for greater clarity in describing the ablation study.

Concern about computational cost.

A: Thank you for raising this intriguing question.

• In response, we have added a group of experiment to Appendix B.6, specifically in the 9th paragraph, titled "When the output resolution of the encoder module increases, how much does it impact the computational cost?"
• Our findings are as follows: During inference, as the resolution of the encoder module's output increases from $32\times32$ to $48\times48$, and then to $64\times64$, the computational time (s) for the baseline without physics loss increased by approximately 92% and 65%, respectively. In comparison, PICL saw an increase of about 83% and 69%. The growth in computational cost for both methods is quite similar. During training, the baseline without physics loss increased by about 277% and 79%, while PICL increased by approximately 238% and 48%. Although PICL's training time increased, its computational cost grew less than the baseline with the rise in resolution. In summary, when using PICL for inference, the computational cost considers only the time for inference. Hence, the efficiency of PICL is essentially comparable to the baseline without physics loss, as PICL does not require computation of physics loss during inference.

Thank you for your answers. Given the contributions of the paper as well as the willingness of the authors' to provide explanations and a considerable amount of additional experimental results, I raised my score to 5. While most of my concerns were addressed, it is still unclear the necessity that we learn surrogate models able to perform low-resolution simulation. The authors' assumption that only coarse-grained data is available in real-world applications is reasonable, but I still doubt that this assumption also motivates one to have low-resolution surrogate models, because in real applications high-fidelity simulation are much in demand.