ClimODE: Climate and Weather Forecasting with Physics-informed Neural ODEs

Thank you very much for many excellent suggestions! We have acted on all of these, and additionally, address all your questions and comments below.

Limited Performance and Comparison: The proposed model, although surpasses vanilla Neural ODE and ClimaX, still cannot compete with IFS (ECMWF NWP), let alone bigger models such as FourCastNet and Pangu-Weather that have already reported better results than NWP. There lacks enough comparison to other models in general, such as Adaptive Fourier Neural Operator (practiced by FourcastNet), previous weather forecasting methods like [1, 2], even models for similar tasks such as spatio-temporal traffic forecasting or video forecasting, etc. Comparison to Neural ODE is more of an ablation study and ClimaX is a weak baseline marginally better than ResNet. Comparing only with ClimaX is definitely not convincing enough.

Comparison with FourCastNet. Thank you for the suggestion. We could not compare to PanguWeather (no public code, model files do not support our data resolution) or GraphCast (public repository seemed incomplete). However, we include an additional experiment compared to FourCastNet. To have comparable results, we re-trained the FourCastNet on our dataset and used their recommended training regime and hyperparameters.

Our results reinforce the efficacy of ClimODE. We note that our method is more accurate in terms of RMSE (in fact, quite significantly especially for the geopolitical variable z) and largely competitive in terms of ACC. We will include these results in the paper.

Thanks for the update. Yes, I totally agree upon the motivation to incorporate physical mechanism. The comparison with FourcastNet seems very promising. Based on that, I would love to raise my score. It seems that when predicting steps increase, the performance drops to a level closer to ClimaX, so the long-term prediction could be less effective.

Thanks so much for your thoughtful comments and excellent suggestions! We've acted on all of them, and also address all your concerns, as we describe below.

"Closed system assumption implies value-preserving manifold" is unclear and needs more justification. While the conclusion sounds intuitive, it's not justified rigorously. The expectation is taken with respect to which density? I would expect the quantity to be preserved would be $\int_{\Omega} u_{k}(x,t)dV$, since that's the total quantity over the whole "volume" you're considering which should be preserved (e.g. when the integral over the volume becomes the mass). In any case, since this is a Machine Learning submission, it would be better if this part is thoroughly justified.

Thanks for pointing this out. Indeed, our expectation notation is imprecise, and we should have written a spatial integral $\int u_k(\mathbf{x},t) d\mathbf{x}$ as you suggest. We will fix this in the paper. We also analysed our model, and found out that the total spatial value is practically constant over time over the ODE forward solver (it shows a flat line over time). We will include this plot in the appendix.

Plot: https://postimg.cc/w709xD9n

Benchmark only up to 36 hours while state-of-the-art methods reported results for up to 10 days ahead. The paper would be better if it reported results for at least five or seven days ahead.

Benchmarking with longer lead times. Many thanks for the suggestion. Based on your feedback, we conducted additional experiments. We show below in the table the RMSE and ACC results for 6-day lead time, and also include 3-day prediction as a reference. We also include ClimaX predictions for comparison. These results show that temperature and potential (t,t2m,z) are relatively stable over longer forecasts, while the wind direction (u10,v10) becomes unreliable over longer times. We note that while ClimaX is also remarkably stable over long predictions, it has lower performance compared to ClimODE. We will include this table in the appendix.

Authors claim that the value-preserving manifold (that emanates from the closed system assumption), presents a strong inductive bias for long-term forecasts, yet, using Euler scheme to solve the ODE is known to be unstable and it'll especially not conserve the quantity we want preserved, so that inductive bias is no longer enforced. It would be better to include that limitation in the paper and acknowledge that while it is a strong inductive bias, it's hard to enforce in practice. This is further seen from Table 2 which shows that the error does increase dramatically with lead time and that suggests that ClimODE is better at inferring the true physics but not in mitigating the error propagation in long-term forecasts.

Thanks for pointing this out. To empirically study this, we analysed how our current model retains the mass-conservation assumption and computed the integrals $I_{k,t} = \int u_k(\mathbf{x},t)d\mathbf{x}$ over time and quantities. We found out that the value is constant over time up to order of $10^{-12}$. Plot link: https://postimg.cc/w709xD9n

Thus it seems that our choice of second-order ODE with Euler with 1 hour increment is a good compromise in terms of accuracy and efficiency, but we agree that higher precision solvers should be preferred. We hypothesise that Euler is sufficient since our dynamics are relatively smooth (eg. Fig 6): the dynamics $\dot{u}_k$ largely models the spatiotemporal trends in climate, while we offload the day-night fluctuations to the instantaneous emission model.

Many thanks for your thoughtful feedback and suggestions. We address all your comments below.

Weaknesses:

A couple of Figures (e.g Figure 5) would benefit from a longer caption/description

Thanks for the feedback. We will add more details to the captions of fig 3 (the pipeline), fig 5 (ODE ablations) and fig 6 (emissions over time) to make them more descriptive. For instance, fig 5 caption will be expanded to

The effect of including different ClimODE components on RMSE. An ablation showing how enhancing the vanilla neural ODE (blue) incrementally with advection (orange), global attention (green) and emission (red) leads to progressively better RMSE scores. We note that that advection yields most improvement in accuracy, while attention turns out to be the least important.

Questions:

What do you mean by one-shot GAN referring to Ravuri et al., 2021? Is there any pre-training involved

Ravuri et al., 2021 propose a GAN framework to model weather prediction as a density estimation problem, where the network learns to predict the weather state at a fixed increment in time. One-shot emphasizes that their model does not unroll the weather state forward in multiple short time steps, instead taking a single leap. They do not perform pre-training and demonstrate their method by modeling precipitation over local regions.

There is Weatherbench 2 available now, maybe to recent to be included in this submission, but should be mentioned in future work: https://arxiv.org/abs/2308.15560, https://sites.research.google/weatherbench/

Many thanks for the reference. We will include a citation and position appropriately in the paper.

Is the comparison against ClimaX fair? The main idea of ClimaX is to use pre-training, but you state you use all methods without pertaining

Thank you for the opportunity to clarify this. We trained the ClimaX model from scratch using the same dataset as for ClimODE without pre-training to compare how much signal the two methods are able to extract from the same data. Comparing to pre-trained ClimaX would mean comparing to a model that has seen much more data than us, which would give unfair advantage to ClimaX. We acknowledge that pretraining can be effective for a practical weather predictor, and are very keen to leverage pre-training in future with larger scale implementations of our advection ODE model.

The statement "IFS is still far ahead of any deep learning method" doesn't really hold anymore, e.g.: https://arxiv.org/pdf/2307.10128.pdf, Kreislers GNN work should be mentioned here as well:https://arxiv.org/pdf/2202.07575.pdf

Thank you very much for these references. We will add these and update our statements about IFS.

Many thanks for your constructive review. We address your concerns and incorporate your suggestions below.

For the model of FLOW VELOCITY (section 3.2), as we already know $\dot{ \mathbf{v}}_{k}(x,t)$ $=$ $\ddot{u}_k (x,t)$ why we still need to parameterize it? Is it because the computation of $\ddot{u}_k (x,t)$ is too costly? There are also many methods that could be approximated. More discussion or claims are encouraged.

Need to parameterize the flow velocity. Thank you for raising an excellent point. In an advection system the climate quantities $u(\mathbf{x},t)$ are transported (moved and redistributed) around by a flow velocity $\mathbf{v}(x,t)$, which is a vector field that points where the climate `mass' is moving. Given a known starting state $u$, the system evolution is fully determined by the velocity $\mathbf{v}$ by solving eq (2) forward in time.

The velocity field function is generally unknown, and we thus learn it from data by parameterising the the velocity function as a neural network $\mathbf{v}(\mathbf{x},t | \theta)$. We also note that the remark that $\dot{\mathbf{v}}_k = \ddot{u}_k$ was a typo, and in general we cannot solve for $\mathbf{v}$ by, e.g., interpolating the value $u$.

Apologies for any confusion: we have fixed this typo and our model equations remain unaffected.

The model treats the advection PDE as independent for each task or quantity. However, in the intricate tapestry of climate dynamics, various quantities are interdependent. For instance, wind patterns can influence temperature fluctuations. The paper doesn't clearly illustrate how the proposed method addresses these inherent inter-correlations.

Explicating how ClimODE accounts for variable correlations. This is another great point that we elucidate here. We present our equations for a single quantity $k$ at a time (for instance, equation (2)) because the convenient vector calculus notation $\dot{u}_k - \nabla \cdot (u_k \mathbf{v}_k)$ applies only to scalar $u$, and a `batch' version $\dot{\mathbf{u}}$ does not admit elegant notation. Nevertheless, our model is still able to consider the entire state vector $\mathbf{u} = (u_1, \ldots, u_K)$ at the same time, and all $K=5$ climate quantities are affected by each other. This can be seen in equation (4), where the velocity change $\dot{\mathbf{v}}_k = f( \mathbf{u}, \ldots)$ of quantity $k$ depends on all state variables, and thus subsequently $\dot{u}_k$ also depends on all state variables. Thus, for instance, wind can affect temperature and vice versa.

To demonstrate the emerging couplings of quantities (ie. wind, temperature, pressure potential), we plot below the emission model $\mathbf{u}^{\mathrm{pred}}(\mathbf{x},t) \in \mathbb{R}^5$ pairwise densities averaged over space $\mathbf{x}$ and time $t$. These effectively capture the correlations between quantities in the simulated weather states. For example, these densities reveal that temperatures (t,t2m) and potential (z) are highly correlated and bimodal; the horizontal and vertical wind directions are independent (u10,v10); and there is little dependency between the two groups. We will include these plots in the appendix.

Plot: https://postimg.cc/0McVK5r5

Again, thank you so much for your extremely constructive feedback. We hope we have sufficiently addressed your concerns, and would appreciate your stronger support for this paper.