Probablistic Emulation of a Global Climate Model with Spherical DYffusion

We thank the reviewer for their positive reception of our work and its "important advances". We are especially glad to hear that you think we have done "a good job at making climate modeling concepts accessible to a wider ML audience." We sincerely hope that, similarly to ML-based weather forecasting, we will see a flurry of research on this in the coming years at top ML conferences.

W1.1: That is a great point. The main reason for using MC-dropout is its simplicity and the fact that the original DYffusion framework also uses it with promising results. You are right that MC-dropout alone does not lead to a skillful ensemble, but the inclusion of the DYffusion framework is key. ACE-STO, made stochastic with MC-dropout in the same way as our method's interpolator network, did not show improvement over deterministic ACE (we also verified that stochastic depth does not improve the MC-dropout-only version of ACE-STO). A possible explanation is that while MC-dropout is crucial for stochastic sampling, the forecaster's predictions remain deterministic and are used to initialize the new autoregressive rollout iteration.

We will further clarify these in the updated version.

W1.2: Thank you for bringing this up. This choice was informed by preliminary experiments focused on training a good interpolator network. There, we found the addition of stochastic depth to slightly improve the interpolator’s validation CRPS scores (for the interpolated timesteps 1 to 5) and significantly improve the calibration of the interpolation ensemble based on the spread-skill ratio (averaged across variables from around 0.26 to 0.35). We will discuss this motivation in our revised draft.

W.2: Thank you for the great suggestion. Below, we have added the corresponding spread’s from the baselines. DYffusion reproduces the ensemble spread similarly well as our method, while ACE-STO clearly has troubles reproducing a variability of similar magnitude as the reference.

Q1: We did not see any obvious visual artefacts at the boundaries of the “image”. However, we did observe biases arise in the DYffusion baseline. For example, Figure 9 shows that DYffusion develops clear biases (left column) for the surface pressure, total water path and meridional wind level-7 variables during the first 90 timesteps that are not (less) present in our method (SFNO-based ACE). Interestingly, these biases are not captured by other weather metrics CRPS, RMSE, spread-skill (right columns). We believe that minimizing such biases is crucial for the long rollouts used to evaluate our climate model emulators, which explains DYffusion's poor performance in terms of climate biases.

We sincerely thank the reviewer for their positive reception of our work, particularly noting its organization, clear relation to previous work, and the practical significance of our contribution to climate modeling and climate change.

W1: This is an excellent point. Our paper focuses on standard evaluation assessments for climate modeling. We cannot compute the data likelihood exactly because the diffusion model can be understood as optimizing a variational lower bound. This is complicated further by the DYffusion framework only falling under the umbrella of generalized diffusion models (not a type of Gaussian Diffusion). It is certainly interesting to evaluate the ensemble simulations with probabilistic metrics. In Figure 7, we take a simple approach by evaluating the spread across ensembles of 10-year time-means. Moreover, in Figure 8, we include a comparison of the ensembles of time-means via the Continuous Ranked Probability Score (CRPS), which shows agreement with the ensemble-mean RMSE of the same. However, the CRPS is a pixel-and timestep-wise metric which may not tell the whole story. To our knowledge, more advanced metrics still need to be carefully developed (for example, https://arxiv.org/abs/2401.14657 presents an interesting approach).

We thank the reviewer for their positive reception of our work and viewing it as an “important contribution” that has “a large number of applications well outside the weather and climate field”.

W1: Sincere thanks for bringing this up. Firstly, we plan to tone it down by rewriting to “(...) a gold standard for successful climate model emulation under our simplified setting (see limitations)”. Secondly, we will make clear in our limitations section that a trained emulator should be expected to inherit any biases of the underlying training dataset (here, FV3GFS) with respect to the real atmosphere. We appreciate your insight, which has helped us refine our presentation of results.

Q1: That is an interesting question. Based on training our own method as well as DYffusion baseline, our intuition is that yes, stability would not be hurt by removing the diagnostic variables from the training set, but we haven’t actually re-trained ACE this way. Also, from reading the paper it seems to us that the main motivation for including these variables was to be able to evaluate how well ACE conserves energy and moisture rather than helping stability.

Q2: This is an excellent question that touches on important aspects of stratospheric dynamics. The top model layer in the FV3GFS-derived dataset encompasses 0-50 hPa, lying entirely in the stratosphere. We note that FV3GFS itself does not have a good representation of the QBO. For sudden stratospheric warming, we are unsure ourselves and would have to dig deeply to evaluate this properly. Generally, stratospheric variables seem particularly challenging to learn due to their slower time scales and partial decoupling from tropospheric weather. Indeed, meridional wind and specific total water at level 0 are amongst the few variables where Spherical DYffusion underperforms ACE in terms of RMSE of the time-means. A more detailed analysis of our model's performance on and potential improvements for stratospheric variables could be an interesting direction for future work.

We thank the reviewer for their positive reception of our work and “strongly” supporting its publication at NeurIPS.

W1: That is an excellent point. We did not try running our method for 100 years. While our current simulations cover 10-year segments due to reference data limitations, we recognize the importance of longer-term stability. We are actively working on 100-year inference runs and aim to provide preliminary results during the rebuttal period, with a full analysis to follow in the revised manuscript.

W2: Thank you for the suggestion. We are eager to improve the estimate of the “noise floor”. For consistency with ACE, we follow their approach and complement it with the estimation of the “ensemble noise floor”. We will include a discussion of alternative methods for estimating the 'noise floor' in our revised manuscript, acknowledging the potential for improving the estimate using approaches based on the central limit theorem.

W3: We appreciate your suggestion. Our choice for the CRPS to measure the quality of the ensembles of time-means was mostly due to its popularity in the time series and, especially, weather forecasting literature. We agree that better approaches exist (as also discussed with reviewer u1ux) and will note so in our revised Fig. 8 and corresponding text in the Appendix.

Q1: Thank you for the great suggestion. We have created videos of two random sample 10-year trajectories by Spherical DYffusion and shared it with the AC (since we are not allowed to directly share links here)*. For the analyzed near-surface wind speed variable (a derived variable from the meridional and zonal wind predictions), we see promisingly realistic outputs compared to the validation climate model simulation.We would be happy to include these visualizations in the final paper.

*We have also included the corresponding snapshots as a set of images in our rebuttal PDF (in case the AC is not able to share the videos with you for any reason). If possible, we would encourage you to look at the videos.

Q2: We appreciate this feedback. We will remove the correlation computation from the figure.

Q3: Great points. We use the basic formulation of the CRPS as implemented in the python packages properscoring and xskillscore. We will make sure to clearly state this in our updated draft. Similarly, we did not include the correction factor when computing the spread-skill ratio. Note that for the 25-member ensemble, this correction factor is just 1.0198. We will fix this in our revised draft.

Q4: This is a great point! The training horizon, $h$, is indeed an important hyperparameter. In our work, we briefly experimented with other horizons (3 and 9) at the initial stage of our project, but then decided to stick with $h=6$ for the following reasons: We believe that it sets a sweet spot between being too small and too large. When it is too small (e.g. 3), it immediately reduces the number of sampling steps for DYffusion and our method, since the reverse sampling process directly corresponds to the time steps $t=0, 1, …, h-1, h$ which can lead to subpar performance. If it is too large, we run the risk that predicting $\mathbf{x}_h$ from early time steps (e.g. based on $\mathbf{x}_0$) is too challenging for the forecasting model. Additionally, the DYffusion paper used a similar horizon of $h=7$ for their sea surface temperature forecasting experiments, which is the data that is most similar to ours. We believe that using $h=9$ or similar values close to $6$ would probably work fine too, but we did not have the compute to run ablations to support this (every new choice of $h$ would require re-training two neural networks sequentially). We will add a discussion of this choice to our revised paper, acknowledging its importance and the rationale behind our selection.

Q5: Thank you for the reasonable suggestion, we will adapt our draft to reflect it.