Turbulent Flow Simulation using Autoregressive Conditional Diffusion Models

Metrics beside MSE (Q2):

We do consider a range of physical evaluations besides MSE values. In addition to the physical metrics shown above, we the evaluate deep learning-based LSiM metric from Kohl et al. (see Fig. 4, Tab. 2). Second, we compare temporal statistics in the form of temporal change rates (see Fig. 8, and a range of results in the ablation appendices C.4 to C.9), as well as correlation to the simulation reference (see Fig. 7 on the right). Finally and most importantly, we do already consider a range of spectral evaluations for temporal as well as spatial aspects (see Fig. 6, Fig. 7 on the left, Fig. 11, Fig. 13, and a range of results in the ablation appendices C.4 to C.9).

Problem Setting and Posterior Sampling (Q3):

The key problems we consider are not deterministic, which means that using probabilistic methods like diffusion models are justified. The isotropic turbulence data set $`Iso`$ contains slices of 2D data from a 3D simulation. This means there is a large range of possible solutions for the evolution of a slice, depending on the flow outside this slice. Thus, the prediction of a slice without knowledge of the outside behavior is by definition underdetermined, and as such non-deterministic. The behavior of the transonic flow $`Tra`$ is also a probabilistic problem. It only exhibits statistically stable average behavior, but the evolution of individual states is not deterministic: Since the simulation is performed with a delayed detached eddy simulation (DDES), not all spatial scales are resolved, meaning there is a range of possible physically plausible evolutions starting from a single state. Furthermore, resampling the data from the fine, irregular simulation mesh to a regular, coarse Cartesian grid introduces additional errors and thus uncertainties. As such, the possibility of creating posterior samples has great potential to increase the accuracy and robustness of fluid flow predictions in these settings.

The only problem we analyze that is deterministic is the incompressible wake flow data set $`Inc`$. It is an established baseline across previous work, and the simplest test case we consider, and such not crucial for the insights of our work. Nevertheless, it is interesting to investigate, to show that diffusion models can also work just as well as deterministic models on such cases.

Another aspect to consider is that real-world applications (e.g., particle imaging velocimetry (PIV), which would be a natural next step for future work) are in general probabilistic. This can be due to a range of factors, the most important of which are measurement noise, measurement sparsity, and uncertainty in the estimation of physical parameters.

We hope we addressed the main weaknesses and questions raised above, and as such kindly ask Reviewer 1NQ6 to reconsider their evaluation of our work.

We would like to thank Reviewer 1NQ6 for the comments. In the following, we respond to the weaknesses (W) and answer the raised questions (Q).

Novelty and Contribution (W):

While the novelty of our work with regard to diffusion methods is limited, we would nevertheless argue that our paper makes some other valuable contributions that are highly relevant for the ICLR community: Multiple concurrent methods investigate a similar problem of flow prediction, showing the relevance of the topic, however none of them considers a direct application of DDPMs in detail. We believe this is an important step that should be taken before moving on to more complex methods. To achieve this, we provide a thorough analysis for various tasks of different difficulty, by comparing a range of established baselines to our method, and we use a wide set of physical evaluation metrics. Furthermore, we provide a detailed appendix with additional evaluations and insights. We also comment on this aspect in more detail in the general response.

Simple metrics (W):

We would like to respectfully disagree with the statement that we only investigate simple metrics. As detailed in the following, one core aspect of our work is that a range of different and meaningful metrics are considered. This includes temporal as well as spatial power spectra, correlations, temporal gradient stabilities, as well as the deep learning-based LSiM metric, which is designed for meaningful comparisons of numerical simulation methods. As such, we believe our evaluations paint a detailed picture of the different architectures and training modalities we investigate.

Physical Conservation Laws (Q1):

This is interesting evaluation and we are happy to provide divergence-free errors for our method here. Since our ground truth simulations are downsampled from a higher simulation resolution for training and evaluations, the ground truth data will also not be fully divergence-free. As such, the errors are only meaningful when comparing to the divergence-free errors of the simulation data. Here, we investigate these errors only on the incompressible flow data set $`Inc`$ and the isotropic turbulence $`Iso`$, as the transonic flow $`Tra`$ is compressible, and thus does not exhibit divergence-free flow fields. To calculate the error, we compute spatial divergence values from velocity gradients computed via central differences. The absolute divergence values are averaged across spatial and temporal dimensions, samples, trained models, and data set sequences. The standard deviation is computed across samples, trained models, and data set sequences. As shown below, ACDM achieves very similar divergence-free errors on $`Inc`$ compared to the simulation baseline, and even results in lower errors on $`Iso`$.

For the transonic flow $`Tra`$, we investigate the conservation of mass instead, by comparing the overall mass in the diffusion predictions with the simulation baseline once again. For this, we aggregate volume-weighted densities across spatial and temporal dimensions, samples, trained models, and data set sequences. The standard deviation is computed across samples, trained models, and data set sequences. As shown below, ACDM results in very similar mass compared to the simulation. For comparison we also report the result of the strongest baseline across cases, the dilated ResNet, here:

1. Divergence-free error on $`Inc`_{low}$:

• Simulation: 0.00049 +- 0.00004
• ResNet-dil: 0.00135 +- 0.00068
• ACDM: 0.00062 +- 0.00014

2. Divergence-free error on $`Inc`_{high}$:

• Simulation: 0.00111 +- 0.00003
• ResNet-dil: 0.00128 +- 0.00013
• ACDM: 0.00116 +- 0.00004

3. Divergence-free error on $`Iso`$:

• Simulation: 0.04359 +- 0.00295
• ResNet-dil: 0.07985 +- 0.03149
• ACDM: 0.03069 +- 0.00455

4. Conservation of mass on $`Tra`_{ext}$:

• Simulation: 0.94850 +- 0.00292
• ResNet-dil: 0.94651 +- 0.00792
• ACDM: 0.94757 +- 0.00382

5. Conservation of mass on $`Tra`_{int}$:

• Simulation: 0.90350 +- 0.00611
• ResNet-dil: 0.89958 +- 0.00764
• ACDM: 0.90779 +- 0.00575

We want to thank Reviewer ySpT for reviewing our submission and for the feedback. In response to the mentioned weaknesses (W) and questions (Q), we would like to present arguments to address the raised concerns.

Novelty (W1):

While the novelty of our work with regard to conditional diffusion methods is limited, we would nevertheless argue that our paper makes some other valuable contributions: Taking a direct approach to applying DDPMs to flow prediction is an open point in multiple concurrent methods, and an important step that should be taken first in our opinion. Here, we provide a thorough analysis of different tasks, by comparing various established baselines to our method, and use a range of physical evaluation metrics. We also comment on this aspect in more detail in the general response.

Comparison at Equal Inference Cost (W2.1):

Making comparisons that are equal across every aspect are very difficult for the problems and baselines we consider. Here, we chose to compare models with a similar capacity measured via the number of trainable parameters and similar training memory requirements. As such, the comparison between U-Net and ACDM is fair, as both use the same number of trainable parameters and architecture, but a different training and inference paradigm. Furthermore, the suggestion of unrolling U-Nets at finer temporal steps would potentially worsen their performance due to additional error accumulation across more rollout steps, causing earlier diverging behavior.

Comparison with Unrolled Training (W2.2):

In spirit of the comparison at equal terms mentioned with regard to W2.1 above, it would conceptually not really be fair to compare U-Nets that receive a substantially larger rollout horizon at training time compared to ACDM. As mentioned in our work, an interesting direction for future research is the application of unrolling at training time to the diffusion approach. Thus, the stability gains from the diffusion approach and the training time unrolling are potentially orthogonal and could be combined, justifying a comparison to U-Nets without unrolling.

We agree that the inference costs of a diffusion based setup are high, but we are confident that future work in the domain will quickly be able to improve this performance with techniques like distillation or improved sampling, in the same way as already emerging in the image domain. First steps towards these speedups were already taken in some concurrent work as discussed in the general response. Furthermore, we would like to clarify that unrolling U-Nets does lead to substantially higher training costs as shown in Appendix B.6. Another key benefit of a diffusion-based predictor model is the ability to create posterior samples which unrolled U-Nets are still unable to do.

Temporal Coherence (W3):

This aspect is not a fundamental problem with the conditioning for the diffusion setup. Rather it is an indication of the substantial difficulty and underdetermined nature of the $`Iso`$ data set, as it does not occur on the other test sets we investigated. Furthermore, the stability analysis via temporal gradients is a metric that considers such issues. For example, this is quite apparent for TF-Enc, which shows clear spikes in this evaluation in Fig. 8, indicating problems with temporal coherence issues. For ACDM, no such issues are visible in Fig. 8, meaning the impact of this visual flickering is quite minor. Substantial fluctuations would either appear as spikes in the mean or as unreasonably large or inconsistent values in the standard deviation in this evaluation. Note that deterministic models are fundamentally less susceptible to problems with temporal coherence, as they circumvent the issue by not providing the ability to create posterior samples in the first place. We also observed that this flickering behavior can be reduced further by increasing the number of diffusion steps.

LSiM Definition (Q1):

Thanks for the suggestion, this is included in Appendix C.3 now.

Fig. 4 (Q2):

Thanks for pointing this out, it is fixed.

Spatial Frequency for $`Iso`$ (Q3):

This is shown in Fig. 11 in Appendix C.1. As mentioned in your question, we did observe and describe that ACDM appears to be dissipative on longer rollouts for $`Iso`$.

Argumentation in Appendix C.5 Summary (Q4):

We agree that the formulation here was unclear, and updated it accordingly.

Subscripts in Appendix C.6 Table 5 (Q5):

Thanks for the suggestions, we updated this (and the other tables in the appendix) to be more clear.

CRPS Metric (Q1):

Thanks for the suggestion, this is an interesting metric which is also use by the concurrent DYffusion method. Our initial evaluation choices were based on established methods for flow physics, and we already investigate a broad range of metrics, but we are happy to consider an evaluation with CRPS in a revised version of our work.

U-Net input steps (Q2):

As clarified in the appendix now, we found $k=1$ to work better for direct one-step predictor models in early exploration runs, meaning we use $k=1$ for all U-Net, ResNet, and FNO models. However, the changes between different values of $k$ are marginal compared to differences between architectures.

Design of Fig. 5 (Q3):

Thanks for pointing this out. The top two rows and the row below use different zoom levels, and as such we intended to visually separate them. This is clarified in the axis titles now.

Choices for Latent Size / Hidden Dimension (Q4):

We investigated larger latent spaces up to $L=256$ for the transformer variants in early exploration runs, but they did not fundamentally improve the behavior. We hypothesize that this is due to a trade-off between an accurate reconstruction and a well-behaved latent space that is easy to evolve temporally. While larger latent spaces tend to produce better reconstruction quality in early time steps, smaller latent spaces are more stable and behave better in terms of temporal evolution. We found $L=32$ to be good trade-off for both aspects. The number of hidden dimensions in the FNO is restricted by the overall number of trainable parameters to achieve a fair comparison with a similar parameter count compared to the other models. This also depends on the number of Fourier modes, as additional modes require more parameters as well. We also investigated a model with a larger hidden dimension of size 224 with (8, 4) modes, i.e. FNO-8, but found even worse performance than FNO-16 or FNO-32.

Number of Diffusion Steps during Training (Q5):

The number of diffusion steps during training is always equal to the number of diffusion steps during inference, i.e. by default $R=20$ for $`Inc`$ and $`Tra`$, and $R=100$ for $`Iso`$. This is clarified across paper and appendix now.

Number of Posterior Samples (Q6):

Yes, across evaluations we always use 5 samples for probabilistic methods.

Transformer Details (Q7):

For TF-MGN, we kept the transformer specification in line with the work from Han et al. (2021), and use transformer decoder layers and residual predictions for the latent model. However, we achieved better results by using transformer encoder layers and direct predictions for the next latent state in the latent model, which we included as the TF-Enc variant. The convolutional encoder and decoder architecture (see Appendix B.5) to embed the data into the latent space is identical across TF-MGN, TF-Enc, and TF-VAE.

Remaining Variants in Tab. 3 (Q8):

The remaining combinations are included in Tab. 3 and across Appendix C.4 now.

Formulation in Appendix C.5 Summary (Q9):

We agree that this formulation was unclear, and updated it accordingly.

Performance of Training Noise Variant in Tab. 5 (Q10):

While we would intuitively also expect a relatively clear trend across noise standard deviations, there is of course no guarantee for a direct, linear relationship. However, we assume this case is caused by a statistical outlier (even though this evaluation should be quite stable as it is computed across 3 random model training runs), as the training with training noise seems to exhibit a higher standard deviation across runs than the other methods.

Additional Visualizations of Ablation Models (Q11):

We included additional visualizations for these ablation models in Appendix E now.

Multiple Predictions via Input Perturbations (W6):

We would like to emphasize that perturbing inputs to achieve multiple predictions actually changes the physics and fundamentally differs from posterior sampling. The former essentially measures how the system reacts to changes in the input from perturbations with forcing, while the latter provides a range of possible and likely results for a single input. The underlying assumption in your suggestion requires that the following two distribution are identical: $s^t = f(s^{t-1} + \mathcal{N}(0, \sigma))$ and $P(s^t|s^{t-1})$. Here, $f$ is a deterministic predictor, e.g. a fluid solver, and $s^{t-1}$ is the input solver state. While we do not provide a formal proof here, one example should illustrate that these distributions can be very different: Assuming $\sigma$ is sufficiently large, the input to $f$ will be so strongly perturbed, that applying $f$ could result in basically any $s^t$, even states that are completely impossible given only $s^{t-1}$. This issue is not fundamentally different for smaller $\sigma$.

Benchmark data sets (W7):

We opted for creating our own data sets for this work due to the following reasons: The incompressible wake flow is an established standard setup in the CFD community, and in recent works like Um et al. (2020) and Geneva and Zabaras (2022). For the more complex cases, we wanted to employ non-deterministic flow setups to highlight the strengths of diffusion models in terms of posterior sampling (see our comment to Q3 from Reviewer 1NQ6 for more details), that are not directly available from e.g. PDE-Bench. As mentioned in the paper, we will make our full data sets publicly available to download upon acceptance, in addition to the already provided source code that allows for generating the data. Nevertheless, we fully agree on the usefulness of benchmark data sets in general to achieve comparable results.

Hyperparameters of ACDM (W8):

We adjusted the argumentation with respect to the hyperparameters at the end of the paper. However, we would like the emphasize that our method works out-of-the-box across cases with the given, linear variance schedule (see Appendix B.1) and a high number of diffusion steps. As such, even one additional hyperparameter in the baselines is a drawback. Adjusting the number of diffusion steps is only necessary if inference performance is crucial. Also note that the ACDM performance changes very consistently across different diffusion steps as shown in Appendix C.4. For training noise or the PDE-Refiner method, parameter tuning ranges from crucial to unavoidable, as otherwise these methods do not improve upon a simple U-Net model or even deteriorate results.

Formulation regarding number of diffusion steps (W9):

As suggested, we adjusted the argumentation regarding the number of diffusion steps in the paper.

Presentation of Fig. 4 and Fig. 6 (W10):

Thanks for pointing this out, we updated Fig. 4 and the argumentation for Fig. 6 accordingly.

We express our gratitude to Reviewer wsiC for the detailed review and feedback. Below, we address the mentioned weaknesses (W) and questions (Q).

Novelty (W1):

While the novelty of our paper with regard to existing diffusion methods is limited, we would argue that our paper nevertheless makes a valuable contribution: Exploring the strengths of a direct DDPM-based setup for flow prediction, is an open point in several contemporary methods, that should be taken as a first step in our opinion. For that, we provide a detailed investigation into tasks of different difficulty, compare to various established baselines, and use a range of physical evaluation metrics. We also comment on this aspect in more detail in the general response.

Abstract Formulation and Benefits of ACDM (W2):

We adjusted this claim in the abstract, and fully revised the argumentation regarding limitations of ACDM at the end of the paper. However, we would like to highlight that ACDM does still have the ability to create posterior samples, while the U-Net baselines with training noise or training rollout do not, as discussed in more detail with respect to W6 below.

Inference Speed (W3):

We agree that the discussion in terms of advantages and drawbacks could be made more clear. We updated the corresponding sections in the paper and appendix accordingly. While the inference costs of our diffusion based setup are high, we are confident that future work will be able to substantially improve performance with techniques like improved sampling or distillation, similar to the imaging domain. As mentioned in W4, first steps towards this are already taken in some concurrent work.

Comparison to Concurrent Work (W4):

Thanks for the suggestion, the mentioned paper is an interesting concurrent approach. We included a comparison in more detail in the general response, and discuss the paper and a comparison to ACDM in a revised version of our paper.

Hyperparameter Deviations (W5):

We comment on each of the mentioned hyperparameter choices in more detail in the revised version of the paper. Specifically, for the number of input steps, we found $k=1$ to work better in early exploration runs for direct predictor models, and $k=2$ to be slightly better for the diffusion setup (also see Appendix B.1 and B.2). However, the differences are quite minor and substantially less than the difference between architectures, e.g., ACDM with $k=1$ achieved about $0.1 * 10^{-3}$ worse results on the $`Tra`$ test sets in terms of the MSE ($0.1 * 10^{-1}$ for LSiM). Similarly, employing a Huber loss worked slightly better for the ACDM setup than training with MSE, but did not fundamentally alter the results (also with a result difference of about $0.1 * 10^{-3}$ for MSE, and $0.1 * 10^{-1}$ for LSiM on the $`Tra`$ test sets). Third, all models were trained until the training loss curves were visually converged. This means more complex models like ACDM or the transformer variants required more iterations, while next-step predictor like U-Net or ResNet learn faster. Furthermore, we found that the latter also deteriorate in performance when trained substantially past this convergence point (also see the details in Appendix B.6).

Temporal super-resolution (Q1):

This is an interesting question, however we believe that it is going beyond the scope of our work. Any autoregressive rollout method that is not explicitly conditioned on physical time does not have the ability to achieve a finer temporal resolution compared to the data (including all our baselines). Of course, traditional temporal super-resolution methods like linear or higher-order interpolation between prediction steps, or training a designated interpolator network are an option. This is an interesting direction that other works like the DYffusion paper mentioned in the general response are already targeting.

Real-World Data Sets (Q2):

We do not see any problem with noisy, real-world data for our method. As shown in the paper, it is robust to interpolation as well as extrapolation tasks. We also experimented with running ACDM from a simple mean flow initialization on $`Inc`$ or $`Tra`$ during inference (without any re-training). We observed that ACDM was able to successfully return to a stable, oscillating prediction, without any stability problems, even from such an extreme out-of-distribution test. Thus, noisy real-world data should not fundamentally impact the approach either.

Concurrent Work (Q3):

We provide a detailed analysis compared to PDE-Refiner in Appendix C.9 in the revised version of our paper. To summarize, we find that ACDM has a range of benefits of PDE-Refiner, with the only drawback being the inference performance, as also mentioned in more detail in the general response.

We would like to thank Reviewer JT77 for the detailed review and feedback. In the following, we address mentioned weaknesses (W) and questions (Q).

Novelty (W1):

While the novelty of our approach with regard to DDPM is agreeably limited, we would argue that our paper nevertheless makes a valuable contribution. Exploring a direct adoption of DDPMs to flow prediction is clear gap in the concurrent work, and an important step that should be taken before moving on to more advanced methods. To achieve this, we thoroughly investigate various tasks with different difficulty, while comparing to a range of established baselines using different physically motivated metrics. We also comment on this aspect in more detail in the general response.

Generalization to different resolutions (W2):

We agree that generalizing to different data resolutions is an interesting direction for future work. It is a fundamental problem across the range of neural network architectures we consider in this work, and even fully convolution methods typically struggle to generalize well far outside the training resolution domain.

Accuracy (W3):

While it is correct that ACDM does not beat every baseline on every data set we consider, it is important to keep the bigger picture in mind and consider the following aspects: First, ACDM is within a small margin to the best method across the test sets shown in Tab. 2. This stability is unmatched compared across baselines, all of which perform substantially worse than ACDM on two or more test sets. Especially see the substantial accuracy increase of ACDM on the most complex $`Iso`$ case, compared to all other models. Second, ACDM allows for posterior sampling, while all baselines expect TF-VAE are deterministic. Third, ACDM is clearly more temporally stable than all baselines from Tab. 2, as shown for example in Fig. 8.

Sampling Speed (W4):

We agree that the sampling costs of ACDM in its current form can be steep compared to other methods. However, we are confident that future work will be able to substantially improve performance with techniques like better sampling or distillation, as shown by substantial recent advances in the imaging domain. First steps towards such improvements are already taken in concurrent work, as discussed in more detail in the general response.

Paper Organization (W5):

Thank you for the suggestion, we adjusted the subsection titles accordingly to make this more clear. We will consider a larger reorganization of this section in the next revision of our work.

Different Data Sets (W6):

We agree that further data sets are always an option to increase the scope of a paper, but we would like to highlight that we already evaluate a range of diverse data sets with different complexity. The incompressible wake flow is relatively simple, but a widely established standard setup in the CFD community. The transonic wake flow is fundamentally more difficult due to the formation of shock waves and the complex vortex shedding that has a range of spectral components in addition to the main shedding frequency. Finally, the isotropic turbulence case is based on direct numerical simulation and highly underdetermined due to the sliced evaluation. Thus, it is unlikely that additional data sets would fundamentally help to improve the answer regarding the central question of our paper: "How does a DDPM setup with appropriate conditioning fare on flow prediction?".

Additional Baselines (W7):

We would like to point out that we consider a wide range of diverse baselines, as well as averages over multiple model runs to achieve a stable and meaningful evaluation. We chose established baseline classes like FNO, ResNets, or U-Nets that are very commonly used across problems. Graph networks definitely represent an interesting class of architectures, however, for the Cartesian data sets at hand, they would not provide any benefits. But this is certainly an interesting direction for future work, as we do not see any reason why the positive aspects of diffusion models should not carry over to graph networks.

Fig. 4 (W8):

Thanks for pointing this out, it is fixed in the revised version.

Novelty and Contribution:

It was mentioned across several reviews that our work exhibits little novelty. However we would like to point out that our work does make a valuable contribution to the ICLR community. We believe that a careful adaptation of unmodified techniques from diffusion methods in the field of image generation is actually still missing in the concurrent publications discussed above. Thus, our submission establishes an important baseline and a fundamental first step, that we believe should be taken before moving on to more advanced methods, such as the ones above [1,2]. We would argue, that our work actually highlights the importance of the overall direction and the concurrent methods, filling in gaps that these papers did not investigate in much detail. Furthermore, we provide a detailed benchmark, ranging from commonly used baselines models to very challenging flow problems, with physically meaningful evaluation metrics. We also provide a range of detailed ablation studies and further insights in our appendix, such as different variants of noisy/non-noisy conditioning. Naturally, we will release our source code and data sets to ensure the that others can use our model architectures and datasets as a benchmark.

There is actually a third paper on time-dependent physics problems with diffusion models [3] at NeurIPS 2023, which, however, more strongly differs in focus. Taken together, these works show a strong interest in the direction of diffusion models in scientific machine learning, and we believe that our submission has an important message for the research community: Even a vanilla DDPM with an appropriate conditioning fares very well on flow prediction. To make clear that we are not inventing a new method, we renamed the main method in our paper from "ACDM" to "cDDPM". For clarity, we still refer to the method as "ACDM" in our responses to the reviews, but this change is already reflected in the revised text of the paper (the figures will be updated accordingly for the camera-ready version). We would be very happy if the reviewers could reconsider their evaluations with respect to the novelty and overall contribution of our work.

[1] S. Cachay, B. Zhao, H. Joren, and R. Yu. Dyffusion: A dynamics-informed diffusion model for spatiotemporal forecasting. arXiv, 2023. doi:10.48550/arXiv.2306.01984

[2] P. Lippe, B. Veeling, P. Perdikaris, R. Turner, and J. Brandstetter. Pde-refiner: Achieving accurate long rollouts with neural pde solvers. arXiv, 2023. doi:10.48550/arXiv.2308.05732

[3] B. Holzschuh, S. Vegetti, N. Thuerey. Solving Inverse Physics Problems with Score Matching. arXiv, 2023. doi:10.48550/arXiv:2301.10250