From Zero to Turbulence: Generative Modeling for 3D Flow Simulation

Thank you for your thorough review. We have uploaded a new version with improved wording.

Stochastic process perspective

While (Yang and Sommer, 2023) have applied a DDPM model to a 2D fluid prediction problem, they do not use or mention the stochastic nature of turbulent flows and present their approach in the usual time series forecasting framework, i.e., predicting a future state given a current state. In contrast, we specifically exploit the insight that 3D turbulent flows behave chaotically with many small-scale and short-lived features to propose generative modeling as an alternative form of neural surrogate model for these systems. Furthermore, our dataset goes beyond their data as well as many other datasets in the machine learning community. First, by simulating in 3D, our data exhibits more small-scale behavior than a comparable 2D dataset. Second, the objects in the flow and how they influence the turbulence patterns make our dataset and paper particularly relevant for practical applications.

Q: Why do you introduce the initial state $\boldsymbol{u}\_{0}, p\_{0}$?

We have changed the term $\mathbb{E}\_{t}[\mathrm{p}((\boldsymbol{u}, p)\_{t} \mid \boldsymbol{u}\_{0}, p\_{0})]$ in Section 4 to $\mathbb{E}\_{t}[\mathrm{p}((\boldsymbol{u}, p)\_{t} \mid \boldsymbol{u}\_{0}, p\_{0}, t)] = \mathrm{p}((\boldsymbol{u}, p) \mid \boldsymbol{u}\_{0}, p\_{0})$ to clarify that our model learns to sample from the flow distribution marginalized across time, i.e., approximates the underlying distribution with a stationary distribution.

In Section 4.1, we propose the generative turbulence simulation task by noting that the above marginal distribution $\mathrm{p}((\boldsymbol{u}, p) \mid \boldsymbol{u}\_{0}, p\_{0})$ is independent of the choice of initial state $\boldsymbol{u}\_{0}, p\_{0}$ under the assumption of ergodicity. It may seem unnecessary to introduce the initial state given the ergodicity assumption, however, this is important to convey our method clearly to the majority of readers who are likely most familiar with neural surrogate models in terms of time series modeling.

Thank you for your valuable review.

Q: Which quantity is visualized in Figures 5 and 6?

Figures 5 and 6 show isosurfaces of the norm of the vorticity $\omega$ as in Figure 4, which visualizes the local rotationality of the velocity field, an essential aspect of turbulence. We have updated the paper accordingly.

Q: Would data augmentation improve performance?

Data augmentation could certainly be helpful since the underlying physics also obeys various symmetries. Ultimately, we believe that the most promising direction for future research is equivariant generative models that obey the same symmetries as the Navier-Stokes equation, e.g. translation and rotation equivariance but also uniform motion and scaling equivariance.

Q: Can your approach be applied to more complex scenarios?

In principle, it is possible to encode, for example, phase information to model multiphase flows or use a graph neural network as the denoising model to work with an arbitrary mesh discretization. In this work, we chose single-phase flows with a grid discretization to focus on the question whether generative models are viable neural surrogate solvers isolated from other, orthogonal modeling aspects, such as multi-scale graph neural networks.

Q: Why did you choose 192×48×48 as your grid resolution?

Our choice of resolution balances dataset size, solver performance, and model memory requirements against the small-scale turbulence phenomena important to our work. We picked a small time step of $\Delta t = 0.1\mathrm{ms}$ between snapshots to ensure that it is possible for the autoregressive baselines to learn the short-lived dynamics of the smallest scales in the turbulent flows. However, these choices already result in a dataset of 9 TB. Even a moderate increase in resolution to 256×64×64 would more than double the size and strain our available computing resources. Furthermore, computational requirements grow quickly with resolution for operations like convolutions and attention. The resolution we chose already limited the batch size in training, even though we trained on A100s and chose a U-Net backbone for our model to minimize memory requirements. Even more efficient architectures are an important research area for future applications in 3D simulations.

Thank you for your thoughtful review.

Source code and data

Please find the link to the anonymized source code and a subset of the data in our non-public global reply.

We will release the source code publicly upon acceptance of the paper. Regarding the dataset, we are still looking for a viable hosting solution that can deal with the size of our dataset (9 TB). Alternatively, the accompanying scripts make it easy to generate the data yourself.

Dataset size

Our dataset consists of $27 + 9 + 9 = 45$ simulations for training, validation, and testing, respectively, with each simulation comprising 5000 snapshots, one taken every 0.1 milliseconds for 0.5 seconds. This gives us $27 \times 5000 \approx 10^5$ samples for training minus the ones corresponding to the transition phase from the initial condition to turbulence. The raw data takes up roughly 150GB for each simulation with a further 50GB after preprocessing the data into an HDF5 file. Overall, this puts our dataset at 9TB for all 45 simulations or 2.25TB for the preprocessed data only. We have amended the paper to clarify this in the main text.

Given the significant size of the dataset, further scaling up our experiments is not viable given our hardware constraints.

Generalization

Our training set comprises over $10^5$ samples based on 27 objects representing a large collection of turbulence patterns. The objects are sufficiently varied that the interactions between different components, e.g. corner configurations or shape widths, and their effects on downstream turbulence can be learned and transferred to unseen objects.

Equivariance and mesh geometries

While we are aware of the limitations concerning equivariance and mesh geometries (described in Appendix F), we respectfully disagree with the reviewer that these are fundamental weaknesses of our approach. As our main contribution is to establish generative models as viable neural surrogate solvers for turbulent flows, we opted for the U-Net architecture widely used for 2D and 3D data. However, our approach is not limited to it; any architecture, including equivariant or graph-based ones, can be plugged in as a backbone. We hope that our framework will benefit from future model developments in these areas.

Speed-up

Though GPUs can outperform CPUs in many tasks, it does not mean that CFD solvers benefit from GPUs in the same way. In fact, OpenFOAM has been reported to achieve a performance boost of just 2x on GPUs. Accordingly, a speed-up of 30x is highly relevant in practice regardless of the hardware differences. Further, please note that the performance of our model is constant with respect to the simulation parameters. In contrast, the runtime of the numerical solver depends on the velocity field $\boldsymbol{u}$ because it needs to scale its internal time step $\Delta t$ to keep the simulation stable.

Thank you for your careful review.

Model architecture
As our main aim is to investigate whether generative models are viable neural surrogate solvers for 3D turbulent flows, we have purposefully chosen each component of our model. DDPM is an established, powerful framework for generative modeling in the image domain, which we chose as the basis of our generative model. U-Nets are well known as a capable neural architecture for grid-based data that produce outputs of the same dimension as the inputs while also creating a compressed internal representation through first down- and then up-scaling combined with skip connections. The spatial downscaling is important for our model so that we can apply a transformer at the coarsest scale and facilitate global communication between all parts of the domain without being held back by the quadratic scaling of attention. Consequently, the three components work together as one unit and neither can be meaningfully ablated.

To emphasize the interplay between the components, we have expanded the description of the model in Appendix D. Furthermore, please see our global reply for the project's complete source code.

Domain representation
To focus on our primary research objective of investigating generative modeling for 3D turbulent flows, we need to disentangle it from other complex issues, such as the representational capacity of graph neural networks. Therefore, from the beginning, we decided to work with a regular grid discretization, which has been thoroughly researched in the ML community, particulary in the image domain, with established components like CNNs and U-Nets. As such, we did not test our model on irregular discretizations such as meshes or point clouds.

Invariance to discretization scheme
Since our model is based on convolutional layers through the U-Net and uses a transformer for global communication between cells, it is fully convolutional and, therefore, in principle, able to transfer to regular grids of different sizes. However, as convolutional layers do not use the physical distance information between cells, our model is not transferable to finer or coarser grids than it was trained on, i.e., the grid's physical resolution needs to match the training data. We have added a sentence discussing invariance with respect to discretization schemes to Appendix F.

Boundary layer
In this work, we focus on modeling the downstream turbulence patterns away from the boundary and the object in the flow influences them. Accurately resolving the boundary layer on a regular grid would require a much higher resolution than is feasible given our storage requirements, as our dataset at its current resolution already needs over 9 TB of disk space.