We thank the reviewer for their concerns and have tried to clarify the points raised.

Weaknesses:

The paper's presentation seems somewhat disorganized. The introduction does not clearly articulate the limitations of current weather forecasting methods or how the proposed method addresses these limitations.

In the second paragraph of the introduction we do outline the drawbacks of existing diffusion models for ensemble weather forecasting. Existing models are 1) computationally expensive due to requiring many sequential forward passes through the neural network and 2) have a coarse temporal resolution. As these models are applied autoregressively they tend to accumulate error over time. Increasing the temporal resolution by taking shorter time steps thus results in higher errors. This is remedied by our method (see e.g. figure 6).

Based on the unclear description provided in the paper, the contributions appear insufficient, and the contributions listed in the introduction seem to be something that any weather forecasting model could achieve.

We respectfully disagree. Most existing weather forecasting models are deterministic and can not produce ensemble forecasts that capture the uncertainty in the dynamics, as we do. There are a few probabilistic models available, as we have discussed in our related work section, but they all differ in key aspects from our method as we explain in more detail in the answer below.

2.What specific contributions does the paper make that differentiate it from other weather forecasting models?

Let's revisit the contributions again. We propose a novel method for ensemble weather forecasting built on diffusion that:

1. can generate ensemble member trajectories without iteration,

Our model can generate temporally consistent forecast trajectories without ever seeing its own predictions. This had not been considered before our work and thus, there is no existing probabilistic weather forecasting model that can do this. The benefit of this is twofold. Firstly, the parallelization means that computational costs can be distributed on large computing clusters, reducing the time it takes to generate forecasts. Secondly, and more importantly, since there is no autoregressive steps, the model does not accumulate errors over time. This property allows us to increase the temporal resolution without sacrificing performance, something SOTA models struggle with.

2. can forecast arbitrary lead-times,

Almost all existing models act on a fixed temporal resolution. With our framework, you are much more flexible. For example, you might want very high resolution in the beginning of a forecast but a lower resolution after a few days. With purely autoregressive models, you are forced to iterate on the high resolution forecast. With our framework you have the freedom to change resolution across a forecast trajectory without affecting performance of temporal consistency.

3. can be used together with iteration to improve performance on long rollouts,

Continuous forecasting is effective for forecasting hours to days but can struggle to forecast longer lead times where the correlation is weaker. Autoregressive forecasting excels at long lead times when used with longer (24h) timesteps [1], but comes at the loss of temporal resolution. Our framework generalizes the autoregressive model, allowing both continuous and autoregressive forecasting in a single model. The ARCI method proposed leverages this by iterating on a longer timestep and forecasting the intermediate timesteps using Continuous Ensemble Forecasting.

4. achieves competitive performance in global weather forecasting.

This is demonstrated in the experiments where the ARCI model outperforms all other models at $\leq$ 6 hour resolution. For a general discussion about baselines, see one of the general comments above.

[1] Bi, Kaifeng, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. “Accurate Medium-Range Global Weather Forecasting with 3D Neural Networks.” Nature 619, no. 7970 (July 2023): 533–38. https://doi.org/10.1038/s41586-023-06185-3.

Dear Reviewer PYFz,

The discussion period ends in a little over 24h. We believe that we have addressed all your concerns. Please consider replying to our answers.

Regards,

The authors

We thank the reviewer for their insightful questions and do our best of answering them below.

Major questions

It is a good idea to use the probability flow ODE from the work proposed from https://arxiv.org/pdf/2206.00364 in diffusion models. But what is the assumption behind the customized ODE in Eq. (2)? Why is there a linear noise level in Eq. (2)? What if $\sigma^2$? More explanation would help.

The probability flow ODE does not affect the physical time process, just the sampling of each time step. We have updated the paper to reflect that any noise level $\sigma$ can be used during sampling. The choice of a linear $\sigma(s)=s$ is based on Karras et al. [1] due to its favorable sampling properties. A different noise scheduling, such as $\sigma(s)=s^2$ is entirely possible in this framework and would not change anything about the forecasting properties. In fact, we can use any continuous normalizing flow model which maps Gaussian noise to "data samples" by solving an ODE (including probability flow diffusion models, flow matching, stochastic interpolants) in our framework, but chose to use the model by Kerras et al. for simplicity. We have clarified this flexibility in the revised paper.

[1] Karras, Tero, Miika Aittala, Timo Aila, and Samuli Laine. “Elucidating the Design Space of Diffusion-Based Generative Models.” arXiv, October 11, 2022. https://doi.org/10.48550/arXiv.2206.00364.

From the paper title and section 4 name, "CONTINUOUS" seems to be used to describe Ensemble Forecasting. However, what the authors really want to convey is the temporal continuity, i.e., any arbitrary fine temporal resolution. You may need to think over the rephrasing. Otherwise, it is confusing for readers (at least for me).

We appreciate the concern for confusing terminology. The name continuous forecasting refers to terminology introduced in [2] for learning a forecasting model that can produce forecasts at any time using lead-time as input. This should be contrasted with iterative or direct forecasting where the forecast is iteratively generated or directly predicted with a fixed lead-time model. The name continuous ensemble forecasting refers to the ability to generate ensemble forecasts using continuous forecasting, something never considered using only machine learning before our paper. However, we will spend some time thinking about how domain-specific terminology can be replaced to ease understanding.

[2] Nguyen, Tung, Rohan Shah, Hritik Bansal, Troy Arcomano, Sandeep Madireddy, Romit Maulik, Veerabhadra Kotamarthi, Ian Foster, and Aditya Grover. “Scaling Transformer Neural Networks for Skillful and Reliable Medium-Range Weather Forecasting.” arXiv, December 6, 2023. https://doi.org/10.48550/arXiv.2312.03876.

In Figure 1, do you need to include the 1-hour prediction as the input for the 6-hour prediction? I saw you have explanations in sections 3 and 4. But Figure 1 deserves a more precise description within its title since it is the main framework of your method.

We thank you for pointing this out and have made immediate efforts in fixing this in the updated paper. Understanding that the 1-hour prediction is not needed as input to the 6-hour prediction is crucial and should be made explicit. We have updated figure 1 and its caption accordingly.

It would be better to provide more comparisons with previous SOTA ML models. Even though I know not all models did not release source codes to the public, but some did, e.g., FourCastNet and ClimaX.

We have written a general answer answering many of the questions related to baselines which can be found in one of the official comments.

The authors mentioned the limitation of diffusion models: computationally expensive. I suggest adding more analysis of computational efficiency.

Sampling a 10-day forecast with 6h resolution for a single member from AR-6h takes 32 seconds, but by parallelizing the 6h timesteps in ARCI-24/6h this reduces to 8 seconds. For higher temporal resolution (which is what we focus on) the savings are even larger.

The Heun solver used during sampling uses 40 function evaluations to generate a single forecast. The deterministic baseline would instead generate a forecast in a single evaluation, reducing the 32 seconds to 0.8 seconds for a 10 day forecast. The same argument can be made for a generative latent-variable model, such as Graph-EFM.

We thank the reviewer for their comments and have addressed all of them below.

Weaknesses

Relation to Zheng et al.

The paper lacks references to closely related methods, such as Zheng et al. [1] on fast sampling of diffusion models via operator learning, which shares similarities with the proposed model’s sampling approach. It would be beneficial to clarify how the proposed sampling method diverges from these approaches.

Thank you for bringing this work to our attention. While Zheng et al. considers a different problem, our work share some interesting connections. They consider a neural operator approach to speed up sampling from diffusion models. This allows them to generate the entire diffusion trajectory in parallel, from a single noise sample.

There are some key differences to our work. Zheng et al. considers sampling of static images and do not have a physical time that drives the system forward. Their parallel sampling is done on the diffusion time axis, while we act on the physical time axis. We still solve the probability flow ODE at each time-step, and make no attempt to parallelize this process. This is perhaps best clarified by our updated version of Figure 1, where the work of Zheng et al. is concerned with speeding up the vertical probability flow solver. Our work still uses the method from Karras et al. for this vertical diffusion sampling, but parallelize generation over the horizontal time axis.

Note that we could use the approach by Zheng et al. in our method as an alterative to Karras et al. to reduce the computational cost of the probability flow sampling. Similar things can be noted about changing the diffusion to other generative models that use the probability flow ODE, such as flow matching or stochastic interpolants.

Since the key contributions of our work are independent of the generative method used, as long as it solves the probability flow ODE, we chose the method by Karras et al. for brevity. We have clarified the flexibility in this design choice in the paper.

[1] Zheng, Hongkai, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. “Fast Sampling of Diffusion Models via Operator Learning.” In Proceedings of the 40th International Conference on Machine Learning, 42390–402. PMLR, 2023. https://proceedings.mlr.press/v202/zheng23d.html.

Thank you to the authors for their response. While it addresses some of my concerns, I have decided to raise my score to 5.

We thank the reviewer for their comments and do our best to address questions and weaknesses.

As introduced in Section 4.2, the main idea is to replace the fixed Z with a stochastic process Z(T) with continuous sample trajectories.

Note that replacing $Z$ with $Z(T)$ in section 4.2 is a generalization of the method in section 4.1. The core contribution is instead to use correlated (fixed or autocorrelated) noise instead of uncorrelated noise. This is what allows the trajectories to be continuous.

Weaknesses

W1: there have been exsiting works for nonautoregressive sequence forecasting by diffusion models. However, realted investigation and discussions are missing, e.g.,

[1] NeurIPS'21 CSDI Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation

The reference refers to imputation rather than the prediction task considered in our work.

Weather forecasting vs General time series forecasting

Is there a significant difference between weather forecasting and general time series forecasting?

What is the input size (the number of variables) of the model in the experiments? It seems there are many variables in the weather forecasting task.

Conceptually, weather forecasting is a type of time series prediction, but the fact that we explicitly represent the weather state at multiple spatial locations (with strong spatial dependencies) makes the problem very high dimensional. The ERA5 data we use in our numerical evaluation is using 5 variables for each grid cell in a 32 times 64 grid, resulting in 10 240 variables at each timestep. This is in stark contrast with the weather dataset considered in [2], which contains measurements of 21 variables at one observation station.

Generating 10 minute forecasts up to 1 week in a single step, as done in [2], thus requires predicting 14 112 variables. Applying the method proposed in [2], of predicting the entire time-series at once, in our setting, would require a massive 2 457 600 variables (for 1h forecasts up to 10 days). While this is perhaps manageable, it should be noted that we have studied a very coarse representation of the atmosphere with just a few selected variables for simplicity. In an operational weather forecasting implementation, we would need to use a much higher spatial resolution and many more spatial fields, which means predicting several steps at once would limit scalability too much for it to be useful in practice. As an example of this, the SOTA probabilistic machine learning weather forecasting model [3] uses a high-resolution version of ERA5 in a 1440 times 720 grid with 84 variables per grid point. Essentially all state-of-the-art MLWP models are based on a 2nd order autoregression for the temporal dimension for this reason.

Nevertheless, we thank you for pointing out the lack of references to the general time-series forecasting literature, where the reference [2] is indeed an excellent method for probabilistic time-series prediction. We will update the related work section to more clearly relate our work to the general time-series forecasting when revising the paper.

[2] ICML'23 Non-autoregressive Conditional Diffusion Models for Time Series Prediction

[3] Price, Ilan, Alvaro Sanchez-Gonzalez, Ferran Alet, Tom R. Andersson, Andrew El-Kadi, Dominic Masters, Timo Ewalds, et al. “GenCast: Diffusion-Based Ensemble Forecasting for Medium-Range Weather.” arXiv, May 1, 2024. https://doi.org/10.48550/arXiv.2312.15796.

Did you perform univariate forecasting tasks for each variable?

As pointed out above, the variables in our dataset are essentially "pixel channels" in a spatial grid. Since the data is highly spatially correlated, predicting each point on its own is not a viable option since it would loose all spatial consistency and the spatial information that is needed for making accurate predictions. In fact, it would be similar to attempting to do video prediction based on a single channel at a single pixel individually. There are also strong correlations across the different variables (channels) in the data, and we forecast all of them jointly with one single model.

It is still not very convincing why a stochastic process is better than random Gaussian noises for nonautoregressive generation. Because we can simply represent the forecast outputs as a vector of a length N. Then we can use a Gaussian vector to yield forecasts.

You are absolutely correct that a stochastic process doesn't make sense in the context of [2]. However, when we are restricted to predicting just a single forecast at a time, the stochastic noise process that drives the diffusion sampling process at different lead times is what allows the model to generate temporally consistent trajectories, despite never seeing its own predictions. If we would independently sample Gaussian noise for each lead time we would end up with a set of reasonable predictions, but that do not make sense together as a weather state trajectory.