Better Neural PDE Solvers Through Data-Free Mesh Movers

We sincerely appreciate your mention of the strengths of our paper and your pieces of advice which help us enhance the manuscript.

Suggestion about the mesh tangling.

Answer: Thanks for your suggestion.

• Firstly, as we use the output of DMM to enhance neural solvers' message passing rather than use it as generated mesh for traditional PDE solvers, mesh tangling does not pose a problem in our context.
• Secondly, to address your concern, we also made an additional discussion and exploration on it in Section 6.4 and Appendix C.2. Constraining the Jacobian determinant of $f$ to be non-negative is equivalent to requiring $\Phi$ in our paper to be convex, which is already constrained by $l_{convex}$. The trend of $l_{convex}$ during training is illustrated in Figure 3 in our paper, showing that it can reach a very low level (nearly zero) early in the training process. Additionally, we find that the generated grid points of test data have $l_{convex}=0$. These demonstrate that this constraint is well met.

Question about roles of three losses within the physics loss.

Answer: Thanks for your question. We conduct new experiments to explore how the three losses impact the performance of DMM in Section 6.4 and Appendix C.3. We reduce the weights of the three losses within the physics loss by a factor of ten for training respectively. Results show that when the weight of $l{equation}$ is decreased, the std and range increase the most, suggesting that the MA equation is not adequately satisfied. The result does not deviate significantly from the original when the weight of $l_{convex}$ is decreased, as $l_{convex}$ can already reach a low level as stated in Answer to Re 1. Figure 4 in the paper shows the plotted meshes of decreased $l_{bound}$, implying that boundary conditions are not as well satisfied as before. All the results above conform to intuition.

Question about resolution transformation.

Answer: Thanks for your question. One of DMM's advantages is its ability to generate meshes of varying resolutions during inference.

• We have added a new experiment in Section 6.4 and Appendix C.1. The result shows that DMM can handle data of resolution different from the training data's resolution.
• Please note that since the selected encoder 1 cannot handle varying resolutions, we need to interpolate the state $u$ prior to inputting states of different resolutions. We assess bicubic interpolation and the softmax interpolation as in Eq 16, with training data at $48\times48$ and 2521 resolutions.
• The meshes from the pre-trained DMM are illustrated in Figure 2 in the paper, and associated quantitative results are in Table 7 and 8 in the paper. Results show that the data adheres well to the MA equation regardless of resolution.

Question about how the proposed method performs given unseen data/boundary conditions/physical parameters in equations.

Answer: Thanks for your question. Firstly, our method can generalize to unseen data in a considerable range due to the generalization ability of the neural networks and the physics-informed data-free DMM training. However, we don't think that our current model can generalize to different boundary conditions and physical parameters in equations since we have not taken it into the input of the model.

Suggestion of improving presentation and text.

Answer: Thanks for your suggestion. We have modified our paper according to the reviewers' suggestions and revised the entire paper.

Question about moving the mesh nodes using the moved mesh in the last timestep.

Answer: Thanks for your question. The reason why we can not learn the meshes based on meshes in the last timestep is that the meshs in the last timestep are not known explicitly at the DMM training period, which means that we can not deduce the physics loss in this situation.

Question about cases where the monitor functions give large values near boundaries.

Answer: Thanks for your question. A new visualization of these cases is provided in Appendix D. We choose some states where the maximum value of the monitor function is relatively close to the boundary to generate meshes. Additionally, we have generated new states with maximum values much closer to boundaries and obtained the new mesh. We can see from Figure 7 that DMM can also handle such situations.

Question about why to only show 20x20 meshes for Burgers' equation.

Answer: Thanks for your question. We have added 48x48 meshes to Appendix D. The reason why we show 20x20 meshes previously is that 20x20 meshes are clearer and can help reduce the length of the main text.

Suggestion about typos of Figure 2(a).

Answer: Thanks for your suggestion. We have fixed it and revised the manuscript to make the paper more clear.

Thank you once again for your valuable insights and suggestions.

We have meticulously addressed your concerns. Please let us know if you have any further suggestions.

We greatly appreciate the suggestions from the reviewers, which have enabled us to further enhance our manuscript.

Concerns about the efficiency of DMM.

Answer: Thanks for your question. To make it clear, we have revised Section 6.1 to show that the Data-free Mesh Mover (DMM) is fast in inference.

• The DMM is an operator trained on data belonging to a distribution rather than a single state. Therefore, once training is complete, it requires only a brief inference time when applied to new data from the same distribution.

• Regarding DMM: we have calculated the inference time of the DMM to be 0.001s for the Burgers' equation and 0.088s for the flow around a cylinder, whereas the conventional PDE solver for the Burgers' equation's mesh adaptation requires several tens of seconds [1].

• Regarding MM-PDE: when we use MM-PDE, the extra cost brought by DMM is comparable with the inference time. A comparison of the MMPDE's time with other baselines' is provided in the table below, showing that its time is comparable to other models. And we have added these above results to the new version of our paper in the section of the 'Burgers' equation'.

[1] Song W, Zhang M, Wallwork J G, et al. M2N: mesh movement networks for PDE solvers[J]. Advances in Neural Information Processing Systems, 2022, 35: 7199-7210.

Concerns for the core contribution.

Answer: Thanks for your discussion about the contribution. Please note that our contribution includes design on both DMM and MM-PDE rather than choosing a specific neural network structure for MM-PDE.

• To further demonstrate that our contribution is orthogonal to the neural network structure in MM-PDE, we have conducted an additional experiment using another model architecture Mesh Graph Nets commonly used to handle irregular grids [1]. This is to showcase the universality and novelty of our method. The table below shows the experimental results of the Burgers' equation. From the table, the error of MGN combined with the MM-PDE framework still surpasses MGN, which indicates the effectiveness of our method. The results are added to the 'Burgers' equation' section.

• For clarity, we have revised our conclusion section to make our contribution clearer: As for DMM, we introduce the physics loss to make it data-free, and we design the model architecture, sampling strategy, and training framework to enhance performance. As for the MM-PDE, we propose a framework to improve the base model and have added many design features tailored specifically for the moving meshes, such as the two-branch architecture, the residual cut network, and interpolation networks capable of adaptively generating interpolation weights. Given the dynamic and irregular grid, we thus choose the GNN network structure employed by Brandstetter et al. as our base architecture.

[1] Pfaff T, Fortunato M, Sanchez-Gonzalez A, et al. Learning mesh-based simulation with graph networks[J]. arXiv preprint arXiv:2010.03409, 2020.

Question about only using the current state for moving mesh.

Answer: Thank you for this question. In practice, we use the state of the current time step as input.

• On the one hand, PDE solvers can only see the state of the current time step during the solving process.
• On the other hand, to the best of our knowledge, traditional methods that use r-adaptive meshes also generate adaptive meshes based on the current time step [1, 2, 3].
• However, we also considered the issue you raised before. We may be able to give the mesh mover the ability to predict the future by introducing a surrogate model to provide information about the future. We had originally intended to explore this idea as a topic for future work. Now we have incorporated our thoughts on this issue into the 'Conclusion and Discussion' section of the manuscript.

[1] Budd C J, Cullen M J P, Walsh E J. Monge–Ampére based moving mesh methods for numerical weather prediction, with applications to the Eady problem[J]. Journal of Computational Physics, 2013, 236: 247-270.

[2] Wallwork J G, Barral N, Ham D A, et al. Goal-oriented error estimation and mesh adaptation for tracer transport modelling[J]. Computer-Aided Design, 2022, 145: 103187.

[3] Budd C J, Williams J F. Parabolic Monge–Ampère methods for blow-up problems in several spatial dimensions[J]. Journal of Physics A: Mathematical and General, 2006, 39(19): 5425.

Thanks for your valuable suggestions to help us improve the manuscript.

Question about the runtimes of MM-PDE and other baselines.

Answer: Thanks for your suggestions. We have added the inference time of MM-PDE to Table 1 in Section 6.1 and list it below. We can see that MM-PDE is comparable to GNN. But the time is slower CNN and FNO since the base model architecture GNN is slower.

Question about the overhead DMM requires.

Answer: Thank you for your questions, we have revised Section 6.1 and 6.2 to add the information to the paper for clarity.

• As for the overhead DMM requires, the DMM for the Burgers' equation has a parameter count of 1222738, with a training time of 7.52 hours and memory usage of 15766MB. The DMM for the flow around a cylinder has 2089938 parameters, with a training time of 22.91 hours and memory usage of 73088MB. And the inference of the DMM is 0.001s for the Burgers' equation and 0.088s for the flow around a cylinder, while several tens of seconds are needed for the conventional PDE solver on the Burgers' equation's mesh adaptation. [1] This means that once DMM is trained, the use of it is quite fast and the time exceeds traditional solvers a lot.

[1] Song W, Zhang M, Wallwork J G, et al. M2N: mesh movement networks for PDE solvers[J]. Advances in Neural Information Processing Systems, 2022, 35: 7199-7210.

Question about the physics loss.

Answer: Thank you for your question.

• We have revised Section 6.4 and provided the physics loss of DMM at the beginning and end of the Burgers' equation and flow around a cylinder both below. It is evident that there is a substantial decrease in this loss throughout the training process, indicating that the DMM has been optimized effectively. The result is also listed below.
• The reason we did not previously provide this loss is that we perceived it as not being sufficiently intuitive. Therefore, we proposed the std and range as more intuitive indicators before.

Question about the optimization of DMM.

Answer: Thanks for your question.

• Our choice to combine the Adam and BFGS optimizers follows many works on Physics-Informed Neural Networks (PINNs) that use several rounds of the LBFGS optimizer at the end of the optimization process [1, 2, 3]. However, in practice, we find that the LBFGS optimizer is quite slow and not very accurate, failing to further reduce the loss. Consequently, we employ the more precise BFGS optimizer.
• To address your concern, we also have conduced an additional ablation study to show whether the BFGS is effective. The result is also listed in Table 2 in Section 6.4 and the table above. We can conclude that BFGS indeed results in a further reduction of the DMM loss.
• However, the BFGS optimizer does require more memory, so we only optimize the parameters of the final layer, and we try to increase the width of this layer as much as possible. The current memory usage has already been mentioned earlier: the Burgers' equation requires 15766MB, and the flow around a cylinder requires 73088MB. A single GPU card would not be sufficient to support our training if we optimized the parameters of the last two layers. However, it can be anticipated that optimizing more parameters could potentially improve the optimization results.

[1] Lou Q, Meng X, Karniadakis G E. Physics-informed neural networks for solving forward and inverse flow problems via the Boltzmann-BGK formulation[J]. Journal of Computational Physics, 2021, 447: 110676.

[2] PATRONI R. Towards adaptive PINNs for PDEs: a numerical exploration[J]. 2022.

[3] Pratama D A, Abo-Alsabeh R R, Bakar M A, et al. Solving partial differential equations with hybridized physic-informed neural network and optimization approach: Incorporating genetic algorithms and L-BFGS for improved accuracy[J]. Alexandria Engineering Journal, 2023, 77: 205-226.

Thank you to the authors for the extensive reply and the additional runtimes and experiments, especially adding GeoFNO as a strong recent baseline, and even an additional 3D experiment, although the improvements are less impressive in 3D, nevertheless the proposed framework shows it can outperform the baseline. The quantitative evaluation of MM-PDE is much appreciated, yet it does raise some concern on the overhead for using the approach. The extra results on Adam and BFGS helps understand the reasoning for using both. Even though the improvement is incremental, and it is more fine-tuning, it can be seen as an engineering solution to make the framework slightly better.

My questions have been addressed properly, thank you once again to the authors for their time to making the paper stronger by incorporating the reviewer feedback.