EddyFormer: Accelerated Neural Simulations of Three-Dimensional Turbulence at Scale

We thank the reviewer for their detailed feedback. We appreciate the recognition of the difficulty of the 3D turbulence, as well as the acknowledgment of our strong results.

Regarding your concerns, we have performed multiple ablation studies on the LES/SGS splitting and the SEMConv block, as well as introduced two new baseline models to further strengthen our results. We address each of your points below and will incorporate the corresponding updates in our manuscript.

Q1: Clarifications on how SEMAttn is performed.

The SEMAttn is defined by Eqn. 11, which mixes features on the same local coordinate from different elements (line 149). Therefore, the $M^3$ modes are neither stacked nor projected, but instead treated independently on every collocation point (line 150).

Q2: Clarifications on how SEMConv is implemented.

The SEMConv is implemented by directly evaluating Eqn. 8. Since the input features are also defined on collocation points, the integral can be accurately computed as a weighted dot product using the corresponding Gaussian weights (line 131). Specifically, the kernel is evaluated at each collocation point, and a dot product is taken with the input features, weighted by the quadrature weights. Since our problems have periodic boundary conditions, we do not apply boundary treatment and instead use simple wrapping.

We will revise the manuscript to include these implementation details for clarity.

Q3: Why do we want a spectral convolution if SEMConv operates “in real space”?

While SEMConv operates on $M^3$ collocation points (not $N^3$ tokens as you mentioned), spectral parameterization is still preferred over standard convolutions. Spectral models such as FNO have proven highly effective for modeling spatiotemporal dynamics and are significantly more expressive than conventional conv-nets, even though they also operate on real-space inputs.

To validate this, we conducted an ablation study on Re94 by replacing SEMConv with standard convolutions (see table below). Consistent with the performance gap between ResNet and FNO, the modified model exhibits a notable drop in accuracy.

Table 1

Q4: How is the model optimized?

The model is optimized using relative L2 error. (Re94: line 187 and Eq. 14; KF4: line 219 and Eq. 16).

Q5: Which inputs and outputs are filtered, and how?

The LES stream uses a spectral cutoff filter with k_max modes for both input and output (lines 115–116). We thank the reviewer for catching the typo on line 119; the correct sentence is: “The initial feature is the projection of the unfiltered input.”

Q6: The architecture feels quite arbitrary, and the choices aren't ablated at all. So we don't know if any performance gains are due to SEM, or how much the SEMConv layers matter.

The SEM basis is not just advantageous but essential to our architecture. It enables efficient localized convolutions, and our coarse-mesh attention relies on the locality and interpolation properties of SEM—features not supported by FFT bases.

To assess this design choice, we conducted an ablation on Re94 by replacing SEMConv with an F-FNO layer. The degraded accuracy confirms SEM as a core component. The training speed also shows that SEMConv is more efficient than FFT-based layers.

Table 2

We have already performed an ablation study in Q3 to examine the difference between SEMConv and standard convolution on grid-based inputs, and the results demonstrate that the spectral parameterization in SEMConv is important to model performance.

Q7: Is there actually any advantage to splitting the model into a SGS, LES path?

Our core insight is to model large- and small-scale dynamics separately, as they exhibit distinct statistical properties and demand different modeling strategies, as discussed on lines 25–31 and 53–59. A single-stream approach is suboptimal: the LES stream cannot capture fine-scale features, while the SGS stream lacks the global expressivity for large-scale dynamics. Even with inter-scale information transfer, their distinct architectures serve different purposes and are not identical.

To investigate this, we have performed new ablation studies on Re94 using LES/SGS stream only. The results below show that the LES stream on its own is weaker without the SGS stream, and the SGS stream on its own is slightly worse than its corresponding F-FNO. As a side note, this also suggests that many global modes in FNOs might be redundant in turbulence modeling tasks.

Table 3

To further support the design of our design, we also included TF-Net [2] in our benchmarks (see Table 4 below). TF-Net uses the LES/SGS splitting but still models both streams using standard U-Nets. TF-Net’s original paper already shows that the LES/SGS splitting is beneficial, and here we reproduced their findings. The comparison highlights that our specialized LES and SGS stream designs offer substantial improvements, demonstrating effectiveness of our architecture.

Q8: Can you just use a (bigger) standard Transformer with more tokens?

A long-standing issue of using Transformers for spatiotemporal modeling is the unfavorable quadratic scaling of computation w.r.t. the number of tokens. Many previous works took different approaches to mitigate this issue (see discussion in Appendix, line 468-483). For Re94, the total number of modes required to resolve the flow is of $\mathcal O(10^7)$. Therefore, using a bigger transformer is not viable.

To strengthen our claim, we conducted additional benchmarks on Re94 using the Axial Vision Transformer [1] (AViT). AViT employs axial attention on regular meshes, making it a fair and representative Transformer baseline. As shown in the table below, AViT underperforms compared to both FNO and EddyFormer, indicating that pure Transformer architectures are less effective for capturing turbulent dynamics. This result further supports our design choice to integrate Transformers with spectral-based convolution.

Table 4

Q9: I’m concerned about whether baseline comparisons were performed properly. For example, Transolver claims superior performance on several CFD benchmarks, including to FNO -- but here it performs significantly worse than FNO.

In Tab. 1, we benchmarked several Transformer variants in previous works. They were primarily evaluated on problems with irregular meshes. For instance, Transolver encodes geometric features by grouping similar mesh points into slice-based tokens—a design well-suited for unstructured domains. We believe its stronger performance compared to FNO is due to FNO’s known limitations on irregular meshes.

Additionally, turbulent flows exhibit intricate fine-scale structures that differ substantially from those in prior benchmarks. For example, Transolver was evaluated on FNO’s 2D turbulence dataset, which is relatively less challenging and has already been shown to be well-handled by new models like F-FNO (Fig. 3 in [4]), which we also include as a baseline. In contrast, our 3D turbulence setting presents significantly greater complexity, making it a more demanding benchmark for evaluating model performance.

Q10: It seems in general FNO performs only a bit worse than EF, but then it completely fails on RB/RT instabilities while EF performs very well. Why do you think that is?

We note that in our data-only benchmark (Tab. 1), the one-step errors of FNOs are almost 3x larger than EddyFormer’s. This gap is fairly large, and is very likely to be enlarged during the solution rollout (Fig. 8 shows the model prediction after 30 roll-out steps).

RT is unstable w.r.t. perturbations. Error is amplified in the rollout stage, which affects both training and inference stage. We believe EddyFormer performs well due to its slower error accumulation over time, in contrast to FNO, whose errors grow rapidly and lead to instability during rollout.

Q11: Details on baseline architectures and resolution. How is the reference solution downsampled?

For F-FNO, we use 64 modes, 4 layers with 32 channels. For FNO, we use 16 modes because of size constraint. For ResNet, we use the standard ResNet-18. For all other Transformers, we use their default architecture.

We refer the reviewer to Tab. 3 for details on dataset resolution, as well as other flow parameters. Re94 is recorded on $96^3$, while KF4 is recorded on $256^2$. The reference solution is downsampled using FFT.

Q12: KF4 is almost identical to the dataset used in the Kochkov paper, yet in this paper Kochkov seems to perform 10x worse than a simple ResNet -- this can't be right.

KF4 follows the same problem setting as Kochkov et al., but differs in both dataset and solver. As noted in Fig. 5, their results are based on correcting a $64^2$ finite volume (FVM) solver, which their own paper [3] acknowledges to be less accurate and efficient than spectral methods. We include their result (dashed green line) as a reference only.

In contrast, our results (line 213) are based on correcting a $256^2$ pseudo-spectral solver, which offers much better accuracy and efficiency. For a fair comparison, we also reproduced their setup using a ResNet trained on our high-accuracy dataset by correcting our $256^2$ solver (Fig. 5, solid green line). We will revise the manuscript to clarify this distinction for readers unfamiliar with the background.

[1] arXiv:2310.02994.

[2] arXiv:1911.08655.

[3] arXiv:2207.00556.

[4] arXiv:2111.13802.

We thank the reviewer for their feedback and for recognizing the strengths of our work, including the dual-stream design inspired by LES, and our comprehensive evaluations on turbulence benchmarks. We particularly appreciate the reviewer’s acknowledgment of EddyFormer's improved performance and acceleration on the challenging three-dimensional case.

I think this paper has solid contribution to learning turbulence by neural operators. The main challenges include the large data scale and the highly complex pattern of turbulence. This work achieves both accuracy and efficiency by carefully integrating latest technique of neural operators with the fluid feature. Specifically, the design of local-global architecture matches the LES decomposition which is well-motivated and empirically effective. The experiments are comprehensive with 3 types of challenging problems and are well conducted, e.g., the resolution is up to $256^3$. All results are showing great advantage of the proposed method.

We thank the reviewer for their detailed and constructive feedback. We appreciate the recognition of the novelty and difficulty of the 3D turbulence case, as well as the acknowledgment of our strong results—particularly the high-resolution results and the model’s ability to generalize across domain sizes.

Regarding your concerns, we agree that additional context on prior works, as well as ablation studies on our model architecture, would help clarify our contributions. We have performed additional ablation studies on SEM basis and the LES cutoff frequency, as well as introduced two new baseline models to reflect prior works. We will discuss each point and address them below.

Q1: Advantages of SEM basis over FFT basis. The Legendre vs Chebyshev comparison implies that the basis does not have a big influence.

Thank you for raising this important point. We answer and expand on this question here and also add an ablation experiment. The use of the SEM basis is not only an advantage, but a necessary design choice in our architecture. Specifically:

The SEM basis enables efficient implementation of compact convolutions using local spectral elements, which is essential for modeling localized interactions in complex dynamics. Our design of self-attention over coarse spatial meshes relies on the locality and interpolation properties of SEM—this would be infeasible using global FFT bases.

The comparison of Chebyshev and Legendre basis does not imply that the SEM basis is an arbitrary choice, but instead shows that multiple bases are compatible in the SEM framework.

On the other hand, if we use FFT basis, the model is limited to global basis, and as you mentioned, exhibits bad scaling w.r.t. the number of modes that it tries to resolve. To assess the impact of this choice, we conducted an ablation study by replacing SEMConv with a standard F-FNO layer, named EF (FFT). The results degraded significantly, highlighting that SEM is advantageous over FFT basis, serving as a core element of our architecture. The training speed results also show that SEMConv with a compact kernel ($s=2$ in this case) is more efficient than FFT on the global field.

Furthermore, we also benchmarked Re94 with an ablation model which only has the LES stream. The resulting accuracy is almost identical to EF (FFT), proving that 1) SEMAttn models large-scale dynamics better than F-FNO does, and 2) SEMConv models small-scale dynamics that FFT basis fails to capture.

Table 1

Q2: Why compare to "unstructured" methods like GNOT and Transolver?

We appreciate the reviewer for this important point. GNOT and Transolver are two SOTA Transformers that have been successfully applied to 3D problems. While there are other “structured” models on rectilinear input meshes, e.g., Poseidon and Multiple-Physics Pretraining [1] (MPP), they were only implemented, trained, and evaluated on 2D problems.

To provide an additional comparison, we have adopted AViT [2] (used by MPP [1]) to 3D and benchmarked its accuracy on Re94. We consider AViT a strong and relevant baseline for EddyFormer due to its structured attention mechanism and demonstrated effectiveness on spatially regular data. We set the “patch” size to be $16^3$ for consistency with its 2D counterpart. The results below show that although AViT achieved better performance than other Transformer baselines, its error is still significantly higher than EddyFormer.

Table 2

Q3: A direct comparison with TF-Net is missing.

We appreciate the reviewer for bringing our attention to this work, and we will discuss it in the paper. As you mentioned, the architecture of TF-Net is fundamentally different from our approach. Most importantly, TF-Net uses three identical conv-nets (at the same resolution) to model the mean and fluctuating components, which do not utilize different properties of these fields.

In FNO’s paper, TF-Net was shown to be outperformed by neural operators. We have also performed additional benchmarking of TF-Net on Re94 to strengthen our argument. The results below show that TF-Net is still worse than F-FNO on three-dimensional problems (F-FNO is best baseline), which is consistent with FNO’s results. In contrast, EddyFormer outperforms both of them by a large margin.

Table 3

We will revise the manuscript to acknowledge TF-Net, discuss its approach to the LES/SGS split, clarify how our architecture differs, and include this direct comparison.

Q4: What are the time steps, and how many rollout steps are evaluated?

The time discretizations for two problems are described on line 178 and line 216, respectively. Re94 uses $\Delta t = 0.5$ and KF4 uses $\Delta t = 1$.

The number of rollout steps are described in Tab. 3 for the training set, and on line 415 for the test set. Re94 is tested for a total trajectory length of $20s$, while KF4 is tested for $50s$. We note that the number of rollout steps is sufficiently large to reach the regime where predictions decorrelate from the ground truth, at which point invariant measures become the more meaningful and reliable indicators of model performance.

Q5: The details of the spectral cutoff for the SGS vs LES split are missing. How is the spectral cutoff chosen, and how much influence does it have?

The spectral cutoff frequency $k_\mathsf{max}$, as well as other architecture parameters, are summarized in Tab. 4 and 5. Re94 uses $5^3$ modes and KF4 uses $4^3$ modes in the LES stream.

The value of $k_\mathsf{max}$ is chosen empirically by balancing efficiency and accuracy. The capacity of the SEMAttn determines the range of large-scale structures that the LES stream can effectively capture. If the spectral filter includes excessively high-frequency modes, the LES stream is unable to fully utilize them due to its limited resolution capacity—particularly under a fixed model size. In such cases, increasing the cutoff frequency adds computational cost without improving performance, as the attention module cannot effectively model these finer scales.

To quantify the impact of the spectral filter size, we performed an ablation study on Re94 by varying $k_\mathsf{max}$ while keeping all other hyperparameters fixed. Specifically, we compared models using both smaller and larger filter sizes than our default value of $5$. The results show that model accuracy increases as $k_\mathsf{max}$ grows up to $5$, beyond which the improvements plateau. This saturation indicates that the LES stream reaches its effective resolution limit around $k_\mathsf{max} = 5$, and including higher-frequency modes does not contribute further—validating our choice of spectral cutoff as both sufficient and efficient.

Table 4

[1] McCabe, Michael, et al. "Multiple physics pretraining for physical surrogate models." arXiv preprint arXiv:2310.02994 (2023).

[2] Ho, Jonathan, et al. "Axial attention in multidimensional transformers." arXiv preprint arXiv:1912.12180 (2019).

We thank the reviewer for their thoughtful feedback and for recognizing the strengths of our work, including our integration of turbulence theory, the dual-stream design inspired by LES, and our comprehensive evaluations on challenging turbulence benchmarks. We particularly appreciate the reviewer's acknowledgment of the improved performance demonstrated by EddyFormer against strong baselines.

We have performed additional energy conservation checks, as well as introduced a new benchmark on domains with irregular geometries. Regarding your concerns on our evaluation and the applicability of EddyFormer, we address each point below.

Q1: Evaluation metrics are limited to relative L2 and don't check if physical quantities are conserved. Since the model also doesn't enforce any hard physical constraints (e.g., conservation laws), it's unclear if conservation is met and how these models will generalize to OOD.

We acknowledge the importance of evaluating conservative physical quantities. We want to highlight that we have already conducted extensive evaluations that go beyond standard error metrics (like L2 relative error) to assess whether invariant physical properties are preserved implicitly. Specifically, we have:

• Energy spectrum. In Fig. 7, we compare the energy spectrum of the model predictions with ground truth, capturing how well the model preserves energy across different scales.
• Structure function. In Fig. 3c and 5c, we evaluate the two-point third-order structure function, a key diagnostic in turbulence that reflects inter-scale energy transfer. As discussed around Eq. 18, this metric directly tests whether the model preserves correct inter-scale statistics, which are tightly linked to conservation laws in turbulent systems.
• Furthermore, since the turbulent flow cases are in statistical equilibrium, evaluating model predictions over long temporal rollouts already implicitly tests for conservation-like invariants. The sustained match over time between predicted and reference statistics indicates that the model generalizes well and does not accumulate unphysical errors.

Nevertheless, we have performed additional energy conservation checks on KF4 and find that EddyFormer is able to preserve energy balance over an extended period of time. This flow problem receives energy injection from the sinusoidal forcing, and dissipates energy at the smallest scales. Specifically, the energy components are:

$

E(t) = \langle \frac 1 2 \mathbf{v}(t, x)^2 \rangle_x, \quad \frac {dE_\mathsf{dis}} {dt} = \nu \langle \nabla \mathbf{v}(t, x)^2 \rangle_x, \quad \frac {dE_\mathsf{inj}} {dt} = \langle \mathbf{f}(x) \cdot \mathbf{v}(t, x) \rangle_x,

$

To test energy conservation, we measure all three components of the energy budget, and record the accumulated energy $E_\mathsf{tot}$, $E_\mathsf{tot}(t) = E(t) + \int_0^t dE_\mathsf{dis} - dE_\mathsf{inj}$, over 50s on the test set. The results below show that EddyFormer is able to preserve the energy balance over an extended period, while F-FNO (the model that we find is second best for this problem, among our benchmarks) shows a significant and systematic dissipation over time.

Table 1

Q2: The evaluation of long-time rollout is limited in duration. If delta t = 1 s, the maximum number of time steps in the examples is at most 50.

Thank you for your comment. We note that our rollout of 50s in KF4 already represents a significantly longer duration than many related works. For example:

FNO reports rollouts of up to 30s, PDE-Refiner adopts a similar temporal range of 15s, APEBench [2], which specifically targets long-rollout evaluation, uses 40 steps at most.

Thus, our rollout length is among the longest in current literature and sufficiently captures the chaotic regime where prediction stability and invariant measures are most challenging.

Q3: The design lacks any specific treatment for flows with wall effects and is unclear how the current architecture will generalize to real-world turbulence with irregular geometries or wall-bounded flows.

This is an important consideration. The SEM framework underlying EddyFormer inherently supports geometric flexibility, making it naturally compatible with wall-bounded flows and irregular geometries. Although our current study primarily focuses on turbulence in homogeneous domains, extending EddyFormer to incorporate explicit wall effects and complex geometries represents a promising direction for future work, leveraging this intrinsic compatibility. We will discuss this point and make it more clear in our updated manuscript.

To test this, we have also performed an additional experiment on acoustic scattering in a maze. This is another subset in The Well, featuring wave reflections in complex geometries. We also follow the training protocol described in The Well, and report the VRMSE scores below.

Table 2

Due to time constraints, we didn’t modify the model to support wall treatment using attention masking, etc. Still, EddyFormer is able to match the performance of the best baseline. The comparisons with FNOs show that the geometric flexibility of SEM is significant.

Q4: The reported speedup is for a specific problem size (256^3) for a specific benchmark. It is unclear how EddyFormer scales at larger sizes and how that compares to DNS scaling. Quadratic scaling in attention costs might limit the scalability of the model at higher resolutions.

We acknowledge the importance of evaluating and extending EddyFormer to larger problem sizes. While our current implementation is tested on $256^3$ resolution, EddyFormer adopts standard attention mechanisms similar to those used in large language models, enabling the use of well-established techniques (such as memory-efficient attention variants and system-level optimizations) to scale to significantly larger contexts. For extreme-scale turbulence problems, additional architectural adaptations, such as localized attention windows or hierarchical representations during training, can further improve scalability. We view this as a promising and tractable direction for future work.

Q5: Data efficiency is not addressed. There is an underlying assumption that there is access to large training data for transformer models but this is not the case in turbulence.

Thank you for this point, we will make the data efficiency aspect more clear, and expand on it here: we note that we use ~100k snapshots to train both KF4 and Re94. For previous works, both FNO and JAX-CFD also use ~100k solution snapshots. We highlight that Re94 is a three-dimensional problem, which exhibits more complex dynamics. Yet, EddyFormer is able to achieve high accuracy and generalization within the 100k data limit.

[1] Lippe, Phillip, et al. "Pde-refiner: Achieving accurate long rollouts with neural pde solvers." Advances in Neural Information Processing Systems 36 (2023): 67398-67433.

[2] Koehler, Felix, Simon Niedermayr, and Nils Thuerey. "APEBench: A benchmark for autoregressive neural emulators of PDEs." Advances in Neural Information Processing Systems 37 (2024): 120252-120310.