AtmosArena: Benchmarking Foundation Models for Atmospheric Sciences

We thank the reviewer for your constructive feedback and for recognizing our work’s consistency, transparency, and reproducibility. We address each specific concern below.

AtmosArena, does not offer a pretraining dataset, only evaluation.

We believe standardizing pretraining data is neither feasible nor desirable - developers of atmospheric foundation models should have the flexibility to select their own datasets, resolutions, variables, and pretraining tasks, as these choices are fundamental to model development. However, all foundation models should demonstrate strong performance across diverse applications. This mirrors established practices in NLP, computer vision, and other scientific domains like protein modeling, where models train on varied datasets but are evaluated against common benchmarks. AtmosArena provides this standardized evaluation framework for atmospheric sciences.

Why do we need AtmosArena?

Existing foundation models, including ClimaX, Aurora, and the recently published Prithvi WxC, all benchmark their models on non-overlapping sets of tasks, making comparisons difficult. AtmosArena aims to provide a single, open-source, multitask framework for benchmarking current and future foundation models, making it both easier and fairer to compare models and assess progress in the field. Having a standardized benchmark also helps improve expandability, since we can keep adding new tasks, datasets, and metrics to AtmosArena for evaluating future foundation models. A similar effort in NLP, the GLUE benchmark, unified diverse language tasks and datasets proposed in other papers into a single evaluation suite, driving significant progress in the field. AtmosArena aspires to bring the same impact to foundation models for climate and atmospheric sciences.

Comparison with ClimateLearn.

AtmosArena extends ClimateLearn significantly in many dimensions:

• More tasks and evaluation metrics: ClimateLearn supports 3 tasks - weather forecasting, climate downscaling, and climate projection, while AtmosArena additionally provides 5 more tasks - S2S forecasting, extreme weather events detection, climate data infilling, atmospheric chemistry downscaling, and air composition forecasting. In terms of evaluation metrics, ClimateLearn uses RMSE, ACC, and Pearson correlation, while AtmosArena additionally supports SpecDiv for forecasting, and IoU, Precision, Recall, F-1 score, and Specificity for extreme weather events detection.
• More datasets: ClimateLearn supports 4 datasets - ERA5, CMIP6, ClimateBench, and PRISM. AtmosArena does not have PRISM, but supports Berkeley Earth for infilling, ClimateNet for extreme weather events detection, CAMS Analysis for air composition forecasting, and GEOS-CF for atmospheric chemistry downscaling. Our ERA5 dataset is also at a higher resolution (1.40625deg) compared to ClimateLearn (5.625deg).
• Much stronger models: ClimateLearn implements three standard deep learning models, Resnet, Unet, and ViT, while AtmosArena includes ClimaX and Stormer, two strong models for atmospheric science.

Comparison with ClimaX and Aurora

AtmosArena supports climate data infilling, extreme weather events detection, and atmospheric chemistry downscaling, which were not considered by either ClimaX or Aurora. In terms of datasets, we additionally support Berkeley Earth for infilling, ClimateNet for extreme weather events detection, and GEOS-CF for atmospheric chemistry downscaling. AtmosArena also provides two standard finetuning protocols to evaluate foundation models that were not studied carefully in previous works.

We thank the reviewer again for the constructive feedback and continued support. We believe the reviewer’s questions and suggestions have significantly improved the paper, and we sincerely hope that the reviewer can take our discussion into account in the final assessment of the paper.

We thank the reviewer for your constructive feedback and for appreciating the large set of tasks AtmosArena offers and the clear writing of the paper. We address each specific concern below.

While you state that in the future you would like to have a public leaderboard, I think this is already something to be expected from such a framework.

We thank the reviewer for this suggestion. We created a leaderboard at https://atmosarena.github.io/leaderboard/ for the tasks we consider in this work. The leaderboard can be expanded in the future to include more models, tasks, and evaluation metrics. We have also added a link to the leaderboard to the paper.

Metrics like Power Spectral Density (PSD), and PSD plots are important and offer more insights than a single error metric value

We thank the reviewer for this great suggestion. We have added the PSD plots for the downscaling task in the updated submission. To create these plots, we computed the 2D Power Spectral Density using the Fast Fourier Transform (FFT) for each spatial field, then performed radial averaging to obtain 1D PSD curves that show how power varies with spatial frequency. For each variable (T2M, Z500, T850), we plotted the PSD curves of the ground truth and predictions from three models (ClimaX, Stormer, UNet) on a log-log scale.

The PSD plots (Figure 2 in the paper) show that all three models show excellent agreement with the ground truth across low to medium spatial frequencies for all variables, indicating they accurately capture large-scale spatial patterns. However, there are notable differences at high spatial frequencies (>0.2): UNet tends to underestimate the power at these frequencies, suggesting it may smooth out fine-scale details, while ClimaX and Stormer better preserve these high-frequency components. The results suggest that two pretrained models, ClimaX and Stormer, have an advantage in preserving fine-scale spatial details compared to UNet.

I think not including any probabilistic metrics across the collected tasks is unfortunate because simple point estimates just come short of the complexities given these tasks

We provided the probabilistic metrics, including CRPS and Spread-skill ratio in the source code. We did not include any probabilistic metrics in the paper only because all models we consider: ClimaX, Stormer, and UNet, are deterministic models. Even the two newer foundation models Aurora and Prithvi WxC are deterministic. We believe this is not a limitation of our evaluation framework but rather a limitation of existing models. We can always benchmark future models on uncertainty metrics if they provide probabilistic predictions.

I would have expected some additional improvements about the shortcomings of those, especially surrounding additional evaluation metrics

In addition to the PSD plots the reviewer suggested, we also provide Spectral Divergence (SpecDiv), a physics-based metric not considered by previous works.

In section 4.5, Table 4, the numbers reported for ClimaX, ClimaX frozen and ClimateBench-NN are exactly the same as the values reported in the ClimaX publication Table 2

Finetuning ClimaX is expensive because we have to finetune one model for each target variable in ClimateBench, so we used the model checkpoints provided by the ClimaX authors as is. Since the model itself is deterministic and we used the same train and test splits as ClimaX, we were able to reproduce the numbers.

Is there a reference for the Spectral Divergence metric you use, or is this a metric you propose in this work?

To the best of our knowledge, Spectral Divergence was first proposed for image analysis [1] and ChaosBench [2] was the first paper that used it for weather and climate.

[1] Chang, Chein-I. "Spectral information divergence for hyperspectral image analysis." IEEE 1999 International Geoscience and Remote Sensing Symposium. IGARSS'99 (Cat. No. 99CH36293). Vol. 1. IEEE, 1999.

[2] Nathaniel, Juan, et al. "Chaosbench: A multi-channel, physics-based benchmark for subseasonal-to-seasonal climate prediction." arXiv preprint arXiv:2402.00712 (2024).

I believe a work like this is highly dependent on the quality of the code

We have submitted the source code as supplementary material. The code has detailed instructions on reproducing the experiments in the paper. We also developed individual components such as models, data, and metrics in a way that can be used independently.

We thank the reviewer again for the constructive feedback and continued support. We spent significant efforts in providing additional metrics, published source code, and the public leaderboard, which we believe have significantly improved the paper. We sincerely hope that the reviewer can take our effort during the rebuttal into account in the final assessment of the paper.

I thank the authors for the additional effort they have put in, especially regarding the leaderboard. However, I have some remaining remarks:

Regarding the PSD metrics, I apologize for being unclear. My point was not specific to climate downscaling as PSD metrics are relevant for all sorts of other spatio-temporal prediction tasks some of which are included in your framework, such as precipitation prediction in climate emulation. My goal is to underline that the evaluation needs to be more holistic than what the Machine Learning lense currently does, predominantly using distance RMSE metrics. For example your updated manuscript regarding Climate Downscaling reads in line 406: " Stormer is the best model in this task with the lowest RMSE and Absolute Mean Bias for most variables, followed by the Unet baseline". But is that the case if you consider the PSD plots? I am not saying that your task is to find the best model, but I would argue that evaluation needs to be a lot more nuanced.

Regarding probabilistic metrics: I understand your point about the models being deterministic. While you argue rightfully that this is a shortcoming of the models, I would also argue that a comprehensive benchmarking framework should "expose" such shortcomings, because a benchmark should not just test what models can already do but set a higher bar. For example Table 1 could include the probabilistic metrics like CRPS with a note that none of the models are able to do probabilistic forecasts. But at the moment the word "uncertainty" appears once in your manuscript, and with no probabilistic metric mentioned or discussion about uncertainty in forecasting or benchmarking, I would argue that this "weakness" remains.

Regarding the quality of the code: Thank you for including the source code in the zip. I did not expect a discussion about code quality but at the minimum I would argue that for a community benchmark framework, it should include docstrings for all functions and classes which seems to be the case for some but not others. I attempted to run the conda installation and code but got:

Pip subprocess error:
ERROR: Could not find a version that satisfies the requirement enrich==0.1.dev82 (from versions: 0.1.dev1, 1.0, 1.1, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 1.2.7)
ERROR: No matching distribution found for enrich==0.1.dev82

failed

CondaEnvException: Pip failed

on Ubuntu 20.04.

I would like to update my score to 5 to reflect the additional work that the authors have put in, however, would not change my score to "acceptance".

We thank the reviewer for your constructive feedback and for appreciating the contributions, well-chosen data, tasks, and baselines of AtmosArena, and the clear writing of our paper. We address each specific concern below.

It isn't entirely clear that any of the presented models are foundation models; it feels like the 'atmosphere' is a bit too specific of a system to qualify as foundational.

What we mean by “foundation models for the atmosphere” is models that can solve various tasks in atmospheric sciences, including forecasting, downscaling, extreme weather events detection, etc. We adopt the term “foundation” similarly to how people have been referring to GPT-x or Llama as language foundation models or Stable Diffusion as an image foundation model. Foundation models are also ubiquitous in other scientific fields such as EVO for genome, Prithvi for Earth science and remote sensing, AstroCLIP for astrophysics, etc. The fact that both ClimaX and Stormer outperform the non-pretrained baseline UNet in many tasks shows their capability of generalizing to different downstream tasks.

The paper could do a better job of explaining why the various tasks are complementary, e.g. because they all rely on some underlying understanding of the covariance of the atmosphere over different spatial and temporal scales (be explicit).

The tasks we chose are complementary from both atmospheric and machine learning perspectives:

• From a domain perspective, atmospheric tasks are broadly categorized into atmospheric physics or atmospheric chemistry, and AtmosArena supports both categories.
• From a machine learning perspective, tasks in AtmosArena are mapped to well-defined problems in machine learning: forecasting, segmentation, super-resolution, inpainting, and counterfactual prediction. As the reviewer mentioned above, these tasks together test a model’s understanding of the atmosphere over different spatial and temporal scales.

We thank the reviewer again for the constructive feedback and continued support. We believe the reviewer’s questions and suggestions have significantly improved the paper, and we sincerely hope that the reviewer can take our discussion into account in the final assessment of the paper.

We thank the reviewer for your constructive feedback and for recognizing the novelty, reproducibility, and diversity of tasks AtmosArena offers. We address each specific concern below.

The authors should add relevant metrics, such as inference FLOPs and model parameters

We have added the inference FLOPs and model parameters of the three baselines (ClimaX, Stormer, and UNet) to the updated submission. The table below summarizes the numbers.

The authors need to provide more evidence (e.g., frameworks and code) to demonstrate the benchmark’s ease of use and reproducibility.

We have submitted the code as supplementary material. The code provides concrete instructions on reproducing the paper's results and using individual components like data and metrics.

Although the authors conducted extensive experiments to show that no single model excels across all tasks, I believe they should further analyze the experimental results rather than simply testing performance across different tasks.

We believe one important aspect of a multitask benchmark like AtmosArena is to compare the performance of different methods across a diverse set of tasks. Through comparisons in diverse tasks and settings, we have also observed interesting and consistent patterns that are useful for future development of foundation models:

• ClimaX tends to perform better in forecasting at longer lead times (S2S scale) than Stormer which is explained by the wider range of lead times used for pretraining ClimaX.
• Multi-source pretrained models like ClimaX tend to perform better at downstream tasks that are significantly different from pretraining, including climate model emulation and extreme weather events detection.
• Freezing the transformer backbone tends to work better in tasks with little data, which demonstrates the transferability of pretrained backbones to downstream tasks.

If you have any specific analysis in mind, please let us know. We are glad to run them during the rebuttal phase.

We thank the reviewer again for the constructive feedback and continued support. We believe the additional metrics and the published source code have significantly improved the paper, and we sincerely hope that the reviewer can take our effort during the rebuttal into account in the final assessment of the paper.