How Far are Today's Time-Series Models from Real-world Weather Forecasting Applications?

Questions

Q1

$L=48$ for $H \in$ {$24,72,120,168$}

The experiment setting is not optimal for long-term time series forecasting (TSF), where transformer models typically excel. For shorter horizons, graph neural networks might be more suitable. The current setting, with input lengths of $48$, may not fairly represent the performance of transformer models, which are designed for larger input lengths (usually $96$ or $336$ in the literature). Including results for different input lengths, especially $96$, would enhance the benchmark.

The current state-of-the-art for long-term TSF is Pathformer. It would be interesting to see its performance on short-term forecasts.

Q2

ISD data from NCEI/NOAA is the same source as the Corrformer and Informer weather datasets. Why are these datasets not included in Table 1? Although Weather from Informer is only one station and Global from Corrformer might have limited weather indicators, it is important to mention them to provide a complete picture and justify why WEATHER-5K should be preferred (cf. clarity section).

Q3

What is the definition of an outlier, and how do the authors differentiate outliers from extreme weather? This distinction is crucial for users to understand modifications and potential model behavior. Have the modified timesteps been flagged for better model interpretation?

Q4

[…] which highlights an unacceptably high RMSE when comparing ERA5 data to real-world observations […]

Are the authors suggesting that ERA5 is inaccurate? To the best of my knowledge, ISD measurements involve both manual and automated processes, which could introduce human error, especially in manual records, which will not be in favor to ISD based dataset (even with errors correction, which might introduce different types of errors).

Q5

Figur 3 e) illustrates the daily temperature variations at Chongqing city in China (57516099999) from June to September in 2022.

How many stations are included in this city? And so, is this relevant to use it as the main example?

Q6

This indicates that ERA5 consistently underestimated the diurnal temperature range at this station throughout the heatwave period.

This observation is based on one station over six months, compared to more than 5 000 stations over 10 years of data. Such a claim requires more extensive analysis. The authors seem to discredit ERA5 while earlier referring it as one of the most accurate real-time forecasting products and using it to fill missing data. Why use ERA5 if it is later discredited? If ERA5 is the most accurate, why not use it directly, especially given the potential for errors in ISD's manual records, extensive work for correction and data coverage limitation?

How many timesteps are considered extreme weather, and what percentage do they represent over the 10-year span? Is this percentage significant enough to promote WEATHER-5K over ERA5?

Q7

In Figure 4a, where is the NWP model? The authors mention the computational cost of NWP models as a limitation but do not provide performance data. The parameters size /cost of some models, like PatchTST, known for high computational cost, seems inconsistent with the literature, which might advocate that the experiment setting (input of $48$) is insufficient to demonstrate predictive power. There is also an issue with the figure: Dlinear should be a point due to the parameter's size of $0.01$ but is not, and there are two circles without text (red and purple). What do they represent? Mamba and Autoformer? What do the training cost and error dotted lines represent?

Q8 - Experiment setting

Early stopping of 3 is, in my experience, too low; 5 or 10 would be more appropriate.

Q9 - Reproducibility

What is the batch size of Corrformer?

Dataset

D1

What is the point of repeating latitude and longitude for each row of a given weather station? This increases the dataset size unnecessarily unless stations are moving.

D2

we have used latitude, longitude, and elevation to represent their geographic locations.

Where is the elevation information? The instance field does not mention elevation in the CSV files.

D3

However, the proportion of error introduced by the interpolation is relatively small.

Authors need to provide the number of errors corrected and a mask or label column to identify these corrected timesteps. This information is crucial for many tasks and could open the dataset to other applications beyond forecasting.

Limitations

L1

Are errors always isolated timesteps on only one variable? If so, interpolation is understandable. If not, especially with consecutive timesteps with errors or missing data on multiple variables, interpolation is limited. In such cases, did the authors use ERA5? If so, make it clearer in the paper, and explain more in details how interpolation from ERA5 are made.

L2

Not all users of this dataset will require worldwide stations for their applications. Have authors provided a way to select a subset of the dataset? If not, please provide explanation of the weather stations name/ID formatting to simplify the selection task.

Clarity / Coherence

C1

Clarify the nature of Weather-5K and the targeted task. In my understanding, it is an hourly multivariate spatio-temporal weather dataset (several stations, each providing time series of several weather indicators) for short-, mid-, and long-term Time Series Forecasting. Forecasting tasks can be done in various fashions, such as:

• Univariate-to-Univariate (U), ex. Predict temperature of station XX for the next day using historical temperature of station XX
• Multivariate-to-Univariate (MU), ex. Predict temperature of station XX for the next day using historical weather indicators of station XX
• Multivariate-to-Multivariate (M), ex. Predict temperature of station XX for the next day using historical weather indicators of station XX
• Spatio-Temporal (ST) U
• ST MU, ex. Predict temperature of station XX for the next day using historical temperature from station AA to station WW
• ST M
• Multi-Variables (V) ST MU
• V ST M

Table 1 does not fully capture the dataset's potential in terms of forecasting tasks. In addition, it should be revised:

• Use commas consistently for thousands.
• If "frequency" and "year" are provided, is "length" necessary? A column for missing data would be more informative.
• For the Exchange dataset, should $8$ be the value for "station" instead of "variable" since each column represents a country?

C2

Task settings

What is the forecasting fashion? M or MU? This is important for reproducibility. If M, are all stations considered, hence the input length of 48 "to balance computation and performance"? it not all stations, which subset of stations is used (for reproducibility)? If MU, which station is the target? And which stations are the inputs (all or a subset)?

C3

While they provide a solid foundation, their performance may be limited when faced with complex patterns or nonlinear relationships.

Do authors have a reference for this assumption?

C4

ML methods […] offer enhanced capabilities to handle nonlinear relationships and complex patterns.

Need references, perhaps [1] and [2].

[1] https://people.math.sc.edu/devore/publications/NLACTA.pdf [2] https://link.springer.com/book/10.1007/978-3-319-58795-0

C5

In section RW, paragraph "Data-driven numerical weather prediction," reintroduce the NWP acronym, the same should be done for GSWF to improve clarity.

C6

For Figures 8 to 14, provide weather station IDs and sample IDs or origin forecast (t) for each row (for reproducibility). In addition, to highlight the necessity of WEATHER-5K, use samples depicting different scenarios, notably including extreme weather, to visually demonstrate model behavior. Therefore, identify and provide cases where extreme weather:

• Is in the input but not the predicted window.
• Is in the predicted window but not the input.
• Is in both.

Furthermore, in these figures, there is no need to repeat the title of each plot and the x-axis; use a 7 x 5 grid with the sharex option.

Proof-read

To cite but a few:

1. [Very important] There is a formatting issue with citations in the first paragraph of the introduction and throughout the paper. Citations should be in parentheses unless the authors' names are part of the text, as per ICLR guidelines:

When the authors or the publication are included in the sentence, the citation should not be in parenthesis using \citet{} (as in “See Hinton et al. (2006) for more information.”). Otherwise, the citation should be in parenthesis using \citep{} (as in “Deep learning shows promise to make progress towards AI (Bengio & LeCun, 2007).”)

2. Thie following text is repeated twice in the paper; remove the unnecessary repetition.

5, 672 weather stations worldwide are selected, spanning the period from 2014 to 2023, ensuring a recent and relevant time frame. This selection process focused on balancing the longevity of station operation, hourly data availabilit y, and the inclusion of diverse weather variables.

Typos:

• “[…] into physically-based NWP models” or “physical-based” should be physics-based, no?
• “(with yea 2022)”
• “[…] benchmark experiments on WEATHER-5K,” comma?
• “[…] perform better. In our benchmarks. So developing efficient time-series […]”
• “[…] are may not be the optimal solution […]”
• “[…] except for Correformer.”
• “Figur 3 e) illustrates the daily temperature […]”
• “[…] of the WEATHER-5 dataset […]”

Q1. How do you view the trade-offs between model complexity and interpretability in practical forecasting applications?

Same as the weaknesses.

None.