IMPLICIT STACKED AUTOREGRESSIVE MODEL FOR WEATHER FORECASTING

1. The innovation is trivial and is not novel at all. All the concepts of memory bank, encoder-decoder structure and lead time embedding are directly from previous studies. In fact, all these concepts refer to old practices in the classical paper of Transformer (https://arxiv.org/pdf/1706.03762.pdf): memory bank is the memory mechanism practiced by RNN & LSTM line of papers, the architecture including the encoder-decoder framework directly inherits from the transformer practice, and lead time embedding is an extension of positional encoding. It is surely okay to use methods from previous papers as long as it is effective, but I hardly think it has anything to do with the weather forecasting task specifically, which is weird. It is okay to propose this model as a general framework, but it would be strange to use the same model (https://arxiv.org/pdf/2303.07849.pdf) to write multiple papers for video, weather, and potentially many other tasks: which objective is the paper really about? Without either technical innovation (substantially new model design for general deep learning framework) or application innovation (model design with in-depth adaption to application, real-world problem definition and problem-solving, real-world setting and data), this paper can hardly position itself. All the data and setting in this paper directly comes from Earthformer, which is a pioneering paper to frame the weather forecasting as a video generation task. Compared to this contribution of Earthformer, this paper only excels in several (not all, since some results are not shown) experiments (please refer to the weakness 2.) following the already defined task.

2. The experimental comparison is quite confusing. The Earthformer paper is certainly worth paying tribute to, but the setting in that paper regarding the video generation and MNIST dataset has already been criticized since it makes the objective unclear: is the paper about weather forecasting or is it about video generation? If it is about weather forecasting, does MNIST dataset matter at all? It even makes the paper less professional about the weather forecasting task. The correct way to do it should be saving everything about video experiments in the appendix, since it has nothing to do with weather forecasting! I would suggest the authors focus more on the benchmark datasets used in real-world weather forecasting nowadays, such as the advanced ECMWF simulation and NWP results (e.g., Operational IFS). Like it or not, NWP is still the main force in the area of weather forecasting. The so-called SOTA results are meaningless if they cannot compete with NWP, or at least be carried out with real-world data settings (long-range prediction, medium-range forecasting, many weather variables to consider and forecast simultaneously, etc.). I would recommend the authors to read papers such as Pangu-Weather and FourcastNet, paying extra attention on how experiments are carried out in their papers (evaluation metrics, data, problem definition, experimental setting, relation to NWP, etc.). Some datasets (excluding WeatherBench) are simply not in the interests of physicists and meteorologists, since short-term forecasting is already quite accurate with existing methods.

3. The baselines compared are too obsolete. Please consider similar settings such as Pangu-Weather, NowcastNet and FourcastNet, or at least more advanced deep learning architectures such as Fourier Neural Operator, Neural ODE, recent/strong Spatio-Temporal/video models. The experimental results for MIMO-VP are represented by "--" in TABLE 3 without explanation. The compared baselines are not consistent in TABLES 1 and 2. At least for TABLE 2, Earthformer (it is the only strong baseline) should not be scratched. If the paper is about weather forecasting, it is important to compare with advanced baselines, unless the paper makes substantial innovation on the technical side (at least having something new in the literature) as a general framework.

4. The presentation need substantial improvement. First, the description of neural computation is not enough. How is the spatial encoder computed and formulated? How is spatial predictor computed? Why are they called “spatial”? Second, in the experiment, how are forecasting steps set for each dataset? How is the MSE calculated: is it the average of the multiple forecasting steps? What are the GPUs used to carry out the experiments? Any reason why Temperature (t2m) Humidity (r) Wind (uv10) Cloud Cover (tcc) are selected? Is the average performance of all weather factors considered?

1. The story-telling of the paper is problematic. The paper states it particularly focuses on the challenges of weather forecasting, and the proposed framework is also for weather forecasting. However, the paper suddenly involves video frame prediction without any context in either the Introduction or Related Work sections. Also, involving results of video frame prediction is also away from the paper's title and focuses on weather forecasting. The large existence of video frame prediction destroys the good presentation of the weather forecasting problem and confuses the reader.

2. The contribution of the work is limited, or at least not well-illustrated. The technical description is only about 1 page (with most content from existing works). The proposed framework simply follows the existing works, say encoder-predictor-decoder and positional encoding, without new innovation or interpretation. The Error-prone Queue & Error-free Queue seem to be relatively novel part of the framework, but if so, the emphasis and interpretation is very limited.

Presentation

Overall, I think the presentation of the paper can be improved a lot. The current writing is unnecessarily complicated in certain sections, while missing some important details.

• The introduction of Error-prone Queue & Error-free Queue is confusing. At first, I thought the model would employ a special mechanism to handle the initial observations and predicted frames differently because of the introduction of these components. However, I learned that the model simply concatenates the observations and predicted frames over the channel dimension, treating them equally. I think introducing these terms is not necessary and removing them would make the paper more understandable.
• Weatherbench-S and Weatherbench-M are introduced but the authors did not mention what version and what resolution they use for benchmarking their model.
• In Table 2, the lead time (how far the model is predicting) is not mentioned.
• In the Component effect section, the dataset used for this experiment is not specified.
• In Table 4, +Stacked Autoregressive is not well explained. Is it equivalent to using initial observations as additional inputs?

In addition, some statements in the paper are vague and not justified or supported by experiments

• The lead time embedding methods cannot guarantee dense spatiotemporal correlation of model outputs --> What do the authors mean by "dense spatiotemporal correlation of model outputs", and why the lead time embedding methods cannot guarantee this?
• By using lead time t as an input, models are capable of generating specific interval forecasts, effectively tackling error propagation --> This is unjustified. Conditioning on the lead time improves the flexibility of the model as it can predict the future at different temporal scales, but since it still uses its own prediction as input, it still has the problem of error propagation.
• Furthermore, this approach facilitates the consideration of correlations between outputs --> What do the authors mean by this?

Significance

My biggest concern is regarding the significance of the proposed method. It is a minor improvement where we simply concatenate the initial observations and the predicted results over the channel dimension, and the authors have only evaluated the models on rather toy-scale datasets and benchmarks.

• In Weatherbench, the authors said the baseline models are typically utilized for video prediction tasks. I found this to be strange. Since the task is weather forecasting, the authors should have compared with state-of-the-art models for weather forecasting, not video prediction. There have been significant advancements in recent years, including FourCastNet [1], ClimaX [2], GNN [3], PanguWeather [4], Graphcast [15], etc., but the paper did not compare with any of these models. Weatherbench is a standard benchmark for weather forecasting, and this should be the main result of the paper.
• The experiments are spread out between weather forecasting and video prediction. While the main focus of the paper is on weather forecasting (given the title and the main text), the experiments seem to put more weight on the video prediction task. This is confusing, because even though there are certain similarities, these 2 tasks are very different. Weather forecasting involves a lot of atmospheric variables compared to RGB channels in videos, and these variables have complex spatiotemporal relationships that are not present in videos. It would be better if the paper could focus on weather forecasting or weather-related tasks and show the model's effectiveness in that setting only.

[1] Pathak, Jaideep, et al. "Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators." arXiv preprint arXiv:2202.11214 (2022).

[2] Nguyen, Tung, et al. "ClimaX: A foundation model for weather and climate." arXiv preprint arXiv:2301.10343 (2023).

[3] Keisler, Ryan. "Forecasting global weather with graph neural networks." arXiv preprint arXiv:2202.07575 (2022).

[4] Bi, Kaifeng, et al. "Pangu-weather: A 3d high-resolution model for fast and accurate global weather forecast." arXiv preprint arXiv:2211.02556 (2022).

[5] Lam, Remi, et al. "GraphCast: Learning skillful medium-range global weather forecasting." arXiv preprint arXiv:2212.12794 (2022).

1. The novelty of this article is to some extent limited. Because I have to say that multi-step autoregressive training is not a particularly novel technique, as this technique has been widely used in [1,2,3].
2. The experiments on the WeatherBench dataset lack some key variables, such as T500 and Z850. I recommend the author highlighting T500, Z850, and T2M variables in the main paper. If author can supplement the multivariate experiments on the WeatherBench, it will make the experiment more complete and comprehensive.
3. At the same time, the author needs to elaborate on how many frames were extrapolated in each experiment.
4. I think the author also needs to provide a detailed description of the impact of error free queue on the experiment. I recommend the author to conduct an ablation experiment, which removes error free queue and comparing the experimental results with only error queue using algorithm 1.
5. I think the autoregressive training method of algorithm1 will consume several times more memories and computational cost than normal training. I think the author needs to conduct in-depth analysis of memories and computational cost.

Suggestions1: This article lacks some recent NWP work: [1]Graphcast: Learning skillful medium-range global weather forecasting. [2]FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead [3]FuXi: A cascade machine learning forecasting system for 15-day global weather forecast

Some suggestions about Algorithm 1: Although I can roughly guess that "e", "d", and "p" represent "encoder", "decoder" and "predictor" respectively, it would be the best to explain them in the annotation of the algorithm. If using "e", "d", and "p" at the same time, it is best to change the "encoder" in the algorithm to "e".