PDE-Diffusion: Physic guided diffusion model for solving partial derivative equations

Major Comments:

1. Diffusion models are conventionally used to model distributions, focusing on generating samples that are similar to the training dataset. In contrast, this paper focuses on a deterministic task: solving a well-defined PDE. How do diffusion models contribute to solving such deterministic problems? Is the sample generated by the reverse SDE consistently close to the deterministic ground truth with small variance? If so, can one replace the reverse SDE with a deterministic process and initial conditions to obtain deterministic and improved results?

2. Although I am familiar with PDEs and diffusion models, I find the methodology section difficult to follow due to unclear notation and inexact mathematical expressions. There are many problems, I just list a few.

(a) In the first paragraph of section 4.2, $\mathbf{D}$ denotes a decoder. According to $\mathbf{D}(z^0, d^i)$, $d^i$ should be a single snapshot in the latent space as residual. However, the next line states $d^i = (d^1_1, \dots, d^i_N)$, consisting of all the intermediate steps in the diffusion process.

(b) What is the meaning of "probability that $d_m^i$ conforms to the governing equations $e$"?

(c) Equations (5)(6)(7)(8) appear to model a conditional diffusion model. However, it is known in the literature that there is no straightforward way to model a conditional diffusion model directly from the original diffusion model without certain approximations (see e.g., the discussion in (Song et al., 2020b)). Since the paper does not clarify the meaning of conditional on $e$, it is unclear what this section is attempting to convey. Equation (6) seems more like an assumption than a mathematical derivation.

(d) Equation (6) involves the notation $\hat{u}^{i-1}_n$ and $\hat{u}^{i}_n$, but the next line discusses the notation $\hat{u}^{n}_i$ and $\hat{u}^{n}_{i-1}$.

(e) What is meant by "derivative kernel" above equation (12)?

3. The paper claims that solving some PDEs takes "months-long simulations utilizing iterative numerical solvers." However, I believe the PDEs tested in this paper are far from such a regime. Solving these PDEs should be relatively quick, perhaps just a few minutes. In contrast, each physical-time evolution step in the proposed PDE-Diffusion model involves multiple diffusion steps, which can be time-consuming, not to mention the extensive training time required for these diffusion models. The authors should provide the solving time of classical solvers, inference time per solution, and total training time for diffusion models to give readers a sense of the time cost.

4. There are placeholders in the paper where results should be reported. For instance, on page 7, it states "PDE-Diffusion has a decrease of x% on average in the PRE metric" and "Our method shows significantly better results in the case of longer prediction window sizes. The MSE decrease is x% (at 2), x% (at 4), x% (at 8), x% (at 16)." These "x%" placeholders indicate hasty composition, suggesting the authors did not even thoroughly proofread the paper.

5. The paper cites (Gao et al., 2022) when introducing the significance of solving PDEs in the first sentence. However, this paper has a very loose connection with PDEs. A keyword search for "PDE" in that paper reveals that the term only appears twice in the related work section. Citing such a work (Gao et al., 2022) with minimal relevance to the paper's focus on PDEs, particularly in the opening sentence, is quite unprofessional.

Minor Issues:

1. What is the "PRE metric"? The paper lacks a definition.

2. In the problem definition, the authors present solving a PDE as finding an approximate solution on a fixed spatial discretization. This is misleading. In the field of numerical PDEs, the choice of spatial discretization is integral to the PDE-solving process, and most numerical methods can accommodate varying discretizations. The authors should clarify that their method is not mesh-free.

3. Typo in the appendix: "which is reasonable when reasonable in the limit of infinite diffusion steps".

4. In several instances, the authors refer to "partial derivative equations." This term is unconventional; Google search results suggest it is rarely used.

• There’s other diffusion-based PDE solver that should be compared to, such as DYffusion [1]
• The baseline is not enough. FNO is not a recent method and not the state-of-the-art model. I recommend authors to compare with more recent neural PDE solvers to better evaluate the effectiveness of the proposed method, such as FFNO [2] and GFNO [3].
• In most rows in table 2 and 3, the MSE and MAE of PDE-Diffusion is larger than the other baselines but is still annotated boldface, which is misleading.
• In section 5.3, many placeholders are not replaced by final results.

[1] Cachay, Salva Rühling, et al. "DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting." arXiv preprint arXiv:2306.01984 (2023).
[2] Tran, Alasdair, et al. "Factorized fourier neural operators." arXiv preprint arXiv:2111.13802 (2021).
[3] Helwig, Jacob, et al. "Group Equivariant Fourier Neural Operators for Partial Differential Equations." arXiv preprint arXiv:2306.05697 (2023).

The methodology part is not clear in cases of how to embed the physics prior $\epsilon_I$ and $\epsilon_B$ during the reverse sampling process. The result doens't support the claim where in many cases, the PDE-Diffusion result is much worse than FNO. The reviewer also has major concern about the writing style. It seems not rigorous and scientific on delivering the message. Details can be found in the questions part.

My most concern with the paper is the soundness of the paper, since I find that there are many errors in the tables in the experiment section.

Errors:

1. The most obvious is in Table 2 and Table 3, where the PDE diffusion has exactly THE SAME results for all the different datasets, and even the same results in Table 2 and Table 3. This is clearly impossible.

2. Table 2 lacks the results for dataset CFD2D turbo2 for FNO, with MAE metric.

3. in Table 1, the bold font should make the best result across all methods. However, CFD2D turbo2, the proposed method is clearly has higher MAE and MSE than FNO.

Other places:

1. in Table 1, DR2D lacks the bold font for any method for 2-step.

2. In section 5.3, in the third last row, it lacks the numbers: "The MSE decrease is x% (at 2), x% (at 4), x%..."

3. In Fig 2, "Double shock" should be "Double Mach"?

These errors and unfinished parts seems be due to the time rush, so the paper is not polished enough. I strongly suggest the authors to double and triple check the paper, to make sure everything is correct, before submitting. I believe that if the paper is properly and carefully done and well-polished, it can be a strong paper. In its current form, I have to suggest rejection.

Besides, it would be great if the authors can analyze why the proposed method outperforms VDM. A comparison with existing work DYffusion [1] may also be preferable. There are also many neural PDE works that evolves the system in latent space, e.g., [2][3][4][5][6]. It may be preferable to discuss them.

[1] DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting, NeurIPS 2023

[2] Multiscale simulations of complex systems by learning their effective dynamics, Nature Machine Intelligence

[3] Learning to Accelerate Partial Differential Equations via Latent Global Evolution, NeurIPS 2022

[4] Latent space subdivision: stable and controllable time predictions for fluid flow, in Computer Graphics Forum

[5] Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders,” Journal of Computational Physics

[6] Deep fluids: A generative network for parameterized fluid simulations,” in Computer Graphics Forum

1. The manuscript is essentially unfinished. There are many typos, missing figures in the table (e.g. Table 2. FNO MAE CFD2D turb2), missing values (e.g. 5.3 Results Analysis: The MSE decrease is x% (at 2), x% (at 4)...).

2. The comparative results show that FNO is better than the proposed. Table 3 shows that except for one or two cases, the PEDM performs worse compared to FNO. In Table 3, the average of the six random initializations with variance should have been reported rather than the individual results of the initialization.

3. The paper fails to compare with more advanced techniques than FNO.

4. The motivation for the use of the diffusion model for solving PDE is not sound. Diffusion is founded on a probabilistic setting while PDE is deterministic and it isn't easy to relate the two.

5. The inference time comparison between the different methods should be shown in comparison. FNO finds the solution in a parallel manner while PEDM does not.