DiffusionPDE: Generative PDE-Solving under Partial Observation

Thank you for agreeing with us that we consider an important problem and propose a technically sound method.

“The main weakness of the method is the limited novelty”

Please see our common response above.

“The experiments do not take into account any stochasticity or uncertainty”

On one hand, we use a deterministic diffusion model. Thus, given partial observations as input, the only stochasticity or uncertainty in our method is the initial random noise. On the other hand, we agree it is important to take this into consideration and hence supplement an experiment where we test the effect of different noise seeds on the initial and final state of Navier-Stokes equations. Please see Figure 6 in the attached PDF. We promise to add this to the final version of the paper.

“The authors need to… drop any time derivatives”

Please see the common response.

Q1: “Can the authors give some intuition why they apply the DPS losses in lines 12 and 13 to the 2nd order correction (line 8) and not apply any trapezoidal rules in this case?”

DPS guides the sampling procedure by the end of each iteration, and we are not specifically modifying it.

Q2: “Are sparse measurements located on a grid?”

Yes, we use a 128*128 grid to represent the different spaces/states of the PDEs and the sparse measurements lie on the grid points. This is a common practice for most works in area [1][2]. However, we can also easily extend our method to be not restricted to grid points by interpolating on the grid and supervising the inerpolated value.

Q3: “It is not clear if DPS works better than classifier-free guidance”

In Figure 8 in the PDF, we evaluate the forward and inverse processes of the non-bounded Navier-Stokes equation, comparing DiffusionPDE with Diffusion with CFG when only initial and final states are considered. Our DiffusionPDE method achieves lower relative errors in both evaluations. We also compare our results with those of Shu et al. [3], where full time intervals are solved autoregressively. In this approach, the error of the final state increases to approximately 13%, which is higher than that of the two-state model (see Figure 10 in the PDF file).

Q4: “Show the number of parameters, training time and inference time for all methods.”

Please see the common response.

[1] Physics-informed neural networks. Raissi et. al. JCP 2019.

[2] Fourier Neural Operator for Parametric Partial Differential Equations. Li et. al. ICLR 2021.

[3] A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstruction. Shu et al. Journal of Computational Physics 2023.

Thank you for your thoughtful comments! We are happy that you find our paper provides diverse experiments and is well-written.

“The technical contribution is limited.”

Please refer to the common response above.

“Some powerful baselines are missing.”

1. U-Net: We trained a U-Net model based on our EDM diffusion model. Initially, we trained the model to map between 500 fixed points in the input space and the full output space (Figure 9 in the PDF). For the Navier-Stokes equation, the prediction of the final state resulted in an average test error of approximately 39.1%, which is significantly higher than the error of our diffusion model. Additionally, we made predictions for the final state using 500 different sampling points, which increased the relative error to approximately 49.2%. Furthermore, we trained another U-Net model to map between 500 random points in the input space and the full output space. This U-Net model resulted in a 101% error during testing, indicating that the model fails to flexibly solve different patterns.

2. OFormer [1]: We examined the forward and inverse process of Navier-Stokes equation for OFormer taking 500 observation points, and the relative errors are approximately 17% and 23% respectively, which are much larger than DiffusionPDE, as shown in Figure 10 in the PDF file.

3. Transolver [2]: We believe that Transolver is applied to 3D geometric PDEs and is not suitable for comparison with our DiffusionPDE without significant modifications to their method.

“Model efficiency comparisons”

Please see the common response.

“time-dependent PDE”

Please see the common response.

Q1: “Figure 4, do both forward and inverse tasks use the same diffusion model?”

Yes. We can do forward and inverse tasks with a single model.

Q2: “So how does DiffusionPDE handle the spatially scattered partial observations?”

We use a 128*128 grid to represent the different spaces/states of the PDEs, so yes the input is still in the regular grid. This is a common practice for most works in area [3][4]. In addition, we leverage DPS [5] to handle the spatially scattered partial observations, which is a technique designed for noisy inverse problems (e.g. inpainting).

[1] Transformer for Partial Differential Equations' Operator Learning. Li et. al. TMLR 2023.

[2] Transolver: A Fast Transformer Solver for PDEs on General Geometries. Wu et. al. ICML 2024.

[3] Physics-informed neural networks. Raissi et. al. JCP 2019.

[4] Fourier Neural Operator for Parametric Partial Differential Equations. Li et. al. ICLR 2021.

[5] Diffusion Posterior Sampling for General Noisy Inverse Problems. Chung et. al. ICLR 2023.

We greatly appreciate your response and recognition of our method. We would like to clarify that DiffusionPDE can utilize continuous coordinates with bilinear interpolation in our prediction space to obtain predicted values for points that are not on the grid. By doing so, for example, the forward problem of the non-bounded Navier-Stokes equation results in a relative error of approximately 6.0%, which is comparable to the scenario without interpolation (approximately 6.9%) and even smaller as more points are considered during the calculation of observation loss, while still remaining significantly lower than OFormer (approximately 16.2%). Overall, the diffusion model is powerful, enabling DiffusionPDE to consistently outperform OFormer. We are happy to mention all these discussions in the revised paper.

Thank you for the detailed and constructive review!

“The method’s success may heavily depend on the strong regularization from the training dataset.”

We use the same data generation methods as other studies [1, 2, 3], ensuring fair comparisons. The process does not favor any specific subset of PDEs. It’s reasonable to assume that a dataset can be obtained containing patterns or regularization about field distributions. For example, factories manufacture parts with material properties following specific distributions, and our method can solve PDEs for these parts. While our method may struggle with out-of-distribution cases, we know of no existing method that performs well under sparse observations without assumptions about PDE coefficients to the best of our knowledge. This is an open and exciting field with great potential.

“The weightings for observation and PDE loss are significantly higher.”

The weights depend on the scale of the coefficients or solutions. In our static PDE dataset, the absolute values of $u$ are much smaller than those of $a$. To ensure their distributions fall between -1 and 1, $a$ and $u$ are scaled differently during dataset preparation. During inference, they are scaled back to their original values to calculate PDE and observation losses. Thus, $\zeta_u$ should be larger than $\zeta_a$ to ensure the accurate recovery of both $a$ and $u$, as $L_u$ is much smaller than $L_a$. For example, in the Darcy Flow equation, $a$ is 3 or 12, while the absolute value of $u$ can be less than 0.01, making $L_u$ approximately $10^{-4}$ times $L_a$, leading to $\zeta_u$ being $10^4$ times $\zeta_a$. Similarly, the choice of $\zeta_{pde}$ is also influenced by the data scale.

Q1(a): “The baseline models… look discontinuous”

All models are trained with full observations. During inference, we apply a mask to set non-observed input values to zero, which can cause output discontinuities, especially with FNO as the model does not guarantee smooth output. Results are smoother after applying the optimization from Question 1(b) (Figure 5 in the PDF). We also considered training on partial observations (Appendix G), where the model maps sparse inputs to full outputs, but this method doesn’t generalize to different sampling patterns. Additionally, using a different PINN codebase with modified hyperparameters [4] results in smoother outputs for the Burgers’ equation (Figure 4 in the PDF).

Q1(b)(c): Suggest optimizing baselines methods

We acknowledge that optimizing the sampling space is another way to run the baselines. We applied this approach, and the results are shown in Table C below and Figure 5 in the PDF. Optimization reduces errors and smooths the solutions. However, the resulting values are smaller due to the smoothing effect from minimizing PDE loss, and the overall error compared to the ground truth remains much higher than DiffusionPDE. This may be due to the difficulty in optimizing the derivatives of noisy $a$ and $u$. Additionally, optimizing FNO and PINO is slow. While other models were optimized within 30 seconds, FNO and PINO took over 10 minutes to converge. Table B above shows these optimized baselines are slower than DiffusionPDE, likely due to the computational complexity of the Fourier transform.

Table C: Relative errors of solutions (or final states) and coefficients (or initial states) when solving forward and inverse problems respectively with sparse observations after optimizing the baselines. Error rates are used for the inverse problem of Darcy Flow.

Q1(c): “Why diffusion process gives better accuracy for PDE problems than other methods like PINNs”

For partial observations, PINNs [1] does not learn any knowledge about the PDE, making it difficult to inpaint the missing parts. Neural operators [2] learn to map the entire coefficient space to the entire solution space and hence have difficulty taking partial observations as input. For full observations, PINNs require iterations of optimization to converge to the solution, which can be vulnerable to local minimal or failure to convergence, leading to higher error. It is also less robust compared to our data-driven method. Our method, on the other hand, enjoys the advantage of combining PDE knowledge and observation guidance through an iterative generative model. Iterative generative models, such as diffusion models, tend to beat feedforward models like GANs.

Q2: “How is the PDE loss computed?”

Sobel filters are applied to the grid with asymmetric padding to compute derivatives. The PDE loss at the boundary is manually set to zero to handle edge effects. This procedure is applied to the pixel space rather than the continuous coordinate space. We promise to supplement these details in the final version of the paper.

[1] Physics-informed neural networks. Raissi et. al. JCP 2019.

[2] Fourier Neural Operator for Parametric Partial Differential Equations. Li et. al. ICLR 2021.

[3] Learning to solve pde-constrained inverse problems with graph networks. Zhao et. al. ICML 2022.

[4] Physics-informed deep-learning applications to experimental fluid mechanics. Eivazi et. al. Measurement Science and Technology 2024.

Thank you for your positive comments on our work! We feel much encouraged that you recognize the novelty of our work.

a fairer comparison with other methods that work with incomplete data and measurements

In addition to GraphPDE, we further compare our method with OFormer [1], Shu et al. [2], and UNet baselines. Please see Figures 7-10 in the PDF.

Q1: data assimilation

We have compared the results using the RBF kernel (Figure 1 in the PDF). For the forward process of solving the Poisson, Helmholtz, and Darcy Flow equations, the RBF kernel achieved solution errors of approximately 14.3%, 23.1%, and 18.4%, respectively, with 500 random observation points. However, when addressing the inverse problem, the errors increased significantly to 141.2%, 143.1%, and 34.0%, respectively. This increase in error is likely due to the inherent challenges of solving inverse problems with such a straightforward method. We are happy to include such discussions in the revision of the paper.

Q2: full time intervals

Please see our reply in the common response above.

Q3: higher dimensions

Yes, we can extend the method to higher dimensions using state-of-the-art diffusion models designed for such cases. For instance, 3D diffusion models (e.g., [3]), especially latent diffusion models, could be appropriate for handling 3D geometric PDEs.

Q4: more challenging PDE examples

To further demonstrate the generalization capability of our model, we conducted additional tests on different data settings for Darcy Flow. In Figure 2 in the PDF, we solve the forward and inverse problems of Darcy Flow with 500 observation points, adjusting the binary values of $a$ to 20 and 16 instead of the original 12 and 3. Our results indicate that DiffusionPDE performs equally well under these varied data settings, showcasing its robustness and adaptability.

Q5: NS and Burgers' equation with much smaller viscosity

Yes, we also test DiffusionPDE on the Burgers’ equation with a viscosity of $1 \times 10^{-3}$ and on the Navier-Stokes equation with a viscosity of $1 \times 10^{-4}$, which are 10 times smaller than the ones in the main paper. For the Burgers’ equation, we are able to recover the full time interval with 5 fixed sensors at a relative error of approximately 6%, which is close to the error of approximately 2~5% in the main paper. For the Navier-Stokes equation, we can solve the forward and inverse problems with relative errors of approximately 7% and 9%, respectively, using 500 observation points. The errors are also close to the ones in the main paper, where the forward and inverse errors of NS are approximately 7% and 10%. Please see Figure 3 in the PDF.

[1] Transformer for Partial Differential Equations' Operator Learning. Li et. al. TMLR 2023.

[2] A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstruction. Shu et. al. JCP 2023.

[3] 3D Neural Field Generation using Triplane Diffusion. Shue et. al. CVPR 2023.