CirT: Global Subseasonal-to-Seasonal Forecasting with Geometry-inspired Transformer

We sincerely thank you for recognizing the idea and results of our work. We have made every effort to faithfully address your comments in the following responses.

Weakness 1 & Questions 1: A detailed explanation of the $1.5^{\circ}$ experimental results of $0.25^{\circ}$ models.

Thanks for raising this concern. We compare model results on latitude-longitude grid points. For all models, we utilize the same approach to transform $0.25^{\circ}\times 0.25^{\circ}$ grid to $1.5^{\circ}\times 1.5^{\circ}$ grid, including obtaining the prediction of FourCastNetV2, Graphcast, and PanguWeather as well as the training grid data of CirT. Specifically, we first obtain the results of $0.25^{\circ}\times 0.25^{\circ}$ models (e.g., PanguWeather), which are represented on a $721 \times 1440$ grid. This grid corresponds to the coordinates $(\lambda, \phi)$ within the domain $\Omega = [-90^{\circ}, -89.75^{\circ}, \ldots, 89.75^{\circ}, 90^{\circ}] \times [-180^{\circ}, -179.75^{\circ}, \ldots, 179.75^{\circ}, 180^{\circ}]$, where $\lambda$ denotes longitude and $\phi$ denotes latitude. Subsequently, we retrieve the results of coordinates $(\lambda, \phi)$ that correspond to the $1.5^{\circ}\times 1.5^{\circ}$ grid, which is represented on a $121 \times 240$ grid within the domain $\Omega = [-90^{\circ}, -88.5^{\circ}, \ldots, 88.5^{\circ}, 90^{\circ}] \times [-180^{\circ}, -178.5^{\circ}, \ldots, 178.5^{\circ}, 180^{\circ}]$.

We can find that CirT outperforms $0.25^{\circ}\times 0.25^{\circ}$ models even is trained on lower-resolution data, verifying its effectiveness. Please see our response to the subsequent question explaining the rationale for choosing a $1.5^{\circ}\times 1.5^{\circ}$ resolution.

Questions 1: In global forecasts, there is currently a $0.25^{\circ}$ prediction model. Is the $1.5^{\circ}$ model necessary?

Thanks for the question. We would like to clarify that global medium-range forecasting and S2S forecasting are distinct. The existing $0.25^{\circ}\times 0.25^{\circ}$ model focuses on medium-range (up to 15 days) forecasting while S2S forecasting generally considers coarse-grained resolutions like $1.5^{\circ}\times 1.5^{\circ}$. Specifically,

• Predictions on the S2S timescale are highly challenging and have long been considered a “predictability desert”. Therefore, to enhance predictability, most methods[1-2] consider a coarse-grained $1.5^{\circ}\times 1.5^{\circ}$ resolution.
• As shown in Table 1 and Figure 3, although $0.25^{\circ}\times 0.25^{\circ}$ model (e.g., PanguWeather) are trained on higher-resolution data, their S2S predictability does not demonstrate superior performance and underform CirT.

Therefore, $1.5^{\circ}$ S2S prediction model is still necessary.

[1] A machine learning model that outperforms conventional global subseasonal forecast models, Nature Communications, 2024. [2] ChaosBench: a multi-channel, physics-based benchmark for subseasonal-to-seasonal climate prediction, NeurIPS 2024.

Weakness 2 & Questions 2: The article lacks experimental conditions for key variables such as wind u and wind v.

Thanks for your constructive comment. We add model comparisons of 10 metre U wind component ($u10$) and 10 metre V wind component ($v10$) in Table 1 of the revised PDF. Note that the retrieved GraphCast model in ECMWF does not include their inference. The results are shown as follows:

Results of Weeks 3-4 prediction:

Results of Weeks 5-6 prediction:

We can find that wind forecasting is more challenging than other comparing variables. For example, Weeks 3-4 t850 ACC of FourCastNetV2 is 0.957 while u10 ACC is 0.830. Under such cases, CirT still performs the best, further verifying its effectiveness.

In addition, we add the following results regarding the wind $u$ and $v$:

• RMSE comparison with data-driven methods w.r.t. different pressure levels of $u$ and $v$ (see Figure 3 of the revised PDF).
• RMSE comparison with numerical methods w.r.t. different pressure levels of $u$ and $v$. (see Figure 3 of the revised PDF).
• Ablation studies results on $u10$ and $v10$ (see Table 2 of the revised PDF).
• $u10$ and $v10$ RMSE comparison w.r.t. latitude (see Table 3 of the revised PDF).

We can observe that these results do not change our original conclusion and contributions.

Dear Reviewer iiXK,

Thanks for your contributions to the reviewing process. As the deadline for the author-reviewer discussion approaches, we kindly request your feedback on whether our responses have satisfactorily addressed your concerns. Should you have any additional suggestions or comments, please do not hesitate to share them with us. We would be more than willing to engage in further discussions and make any necessary improvements.

Thank you once again for dedicating your valuable time to reviewing our work.

Dear reviewer，

Thanks for your valuable comments. As the deadline for revision (closes on November 27th) is fast approaching, we would be grateful if you could allocate some time to review our revision.

We understand that you have a multitude of responsibilities, To facilitate a swift evaluation of our revisions, we have summarized the corresponding changes as follows:

• We add a detailed explanation of how we obtain the results of the Pangu model (Lines 252-260 in the revision).
• We add the experimental results of wind $u$ and $v$ (Table 1, Table 2, Table 3, and Figure 3 in the revision).
• We discuss the difference between CirT and FourcastNet in Section 5 (Lines 523-527).
• We conduct an ablation study to compare the performance of direct training and fine-tuning (Table 5 in the revision).

Please let us know if you have any additional concerns or questions. We kindly request that you re-evaluate our paper based on the provided responses and revision. Thank you for your time and consideration!

Dear Reviewer iiXK,

Thanks for your contributions to the reviewing process. We kindly request your feedback on whether our responses have satisfactorily addressed your concerns. Should you have any additional suggestions or comments, please do not hesitate to share them with us. We would be more than willing to engage in further discussions and make any necessary improvements.

Thank you once again for dedicating your valuable time to reviewing our work.

Dear Reviewer iiXK,

Thanks for your comment. Given the substantial work involved, we would appreciate it if you could confirm whether our response has addressed your concerns. If there are any remaining issues, we are willing to engage in further discussion and make additional adjustments.

We sincerely appreciate the reviewer for listing the detailed strengths and taking the time to provide feedback. Your support of our work is greatly appreciated. We try our best to eliminate the question via the following response.

Weakness & Questions: An in-depth discussion on the model design and system architecture distinctions between GraphCast and CirT.

We thank the reviewer for raising the comparison between CirT and GraphCast. We agree that both GraphCast and CirT leverage geometric inductive biases. Nevertheless, they are distinct and we highlight our technical contributions from the following aspects:

• Motivation: GraphCast focuses on local state aggregation, probably due to its goal is short-term medium-range forecasting. In contrast, CirT aims to forecast weather in the S2S timescale and focus on capturing the global relations between initial and S2S weather states. Thus, CirT is constructed on a transformer foundation that excels at capturing global information.
• Model design: GraphCast relies on message passing to aggregate local information without explicitly accounting for spatial periodicity. In contrast, CirT employs circular patching to normalize patch geometry and leverages its Fourier representation, consisting of coefficients of periodic basis functions, as inputs to the transformer encoders.
• Empirical verification: Extensive experiments in Table 1 and Figure 3 show that CirT outperforms GraphCast on S2S predictability. Additionally, in Table 3, we can find that CirT outperforms GraphCast in different latitudinal areas, further verifying the effectiveness of our model designs.

Dear Reviewer PHHy,

Thanks for your contributions to the reviewing process. As the deadline for the author-reviewer discussion approaches, we kindly request your feedback on whether our responses have satisfactorily addressed your concerns. Should you have any additional suggestions or comments, please do not hesitate to share them with us. We would be more than willing to engage in further discussions and make any necessary improvements.

Thank you once again for dedicating your valuable time to reviewing our work.

We sincerely appreciate the reviewer for listing the detailed strengths and constructive comments. We have made every effort to faithfully address your comments in the following responses.

Weakness 1: Comparison of model computation complexity and model size.

Thank you for raising the comparison of computation complexity/model size. We compare CirT's Floating point operations (FLOPs) and parameters with two representative models, Graphcast and PanguWeather. The results are as follows:

We can observe that CirT has less computation and smaller model size (due to the lower resolution and embedding size), but achieves better S2S predictivity, verifying our model designs.

Weakness 2: Ablation study of autogressive prediction vs directly predicting all the future values.

Thanks for the instructive suggestion. We adapted CirT's output head to forecast next-day weather variables based on the input date. For inference, it iteratively predicts next-day weather variables up to the S2S timescale. The results are displayed as follows:

Results of Weeks 3-4 prediction:

Results of Weeks 5-6 prediction:

From the results, we can observe that the direct method performs better. The autoregressive CirT still accumulates errors, resulting in inaccurate S2S predictions. Additionally, it underperforms models like PanguWeather and GraphCast which are trained on higher-resolution data (i.e., $0.25^{\circ}$). In contrast, the direct CirT trained on the same dataset (i.e., $1.5^{\circ}$) performs the best, demonstrating its effectiveness.

Question: Why the proposed method patch all the different longitudinal areas into the same patch and do e.g. axial transformer with Fourier Transform?

Thanks for the question and reference. We address the question from the following three aspects:

• Due to the inherent geometric differences between the sphere and the plane, the resulting regular 2D patches will have varying areas and spatial relations. In contrast, circular patch lengths can be determined by their latitudes and the adjacent patches are equidistant, reducing the learning difficulty.
• Another inductive bias is that the circular patch is spatial periodic. That is, the circular patch satisfies $X_{w} = X_{w + W}$. Therefore, instead of directly inputting $X$ into the transformer, we consider its Fourier transform, which is composed of coefficients from a series of periodic basis functions.
• Ablation studies in Table 2 show the performance of grid (2D input), grid with Fourier Transform (FT), circular patching, and circular patching with FT. We can observe that circular patching consistently enhances model performance, with optimal results achieved when combined with Fourier transforms, verifying our model designs.

Dear Reviewer pWHL,

Thanks for your contributions to the reviewing process. As the deadline for the author-reviewer discussion approaches, we kindly request your feedback on whether our responses have satisfactorily addressed your concerns. Should you have any additional suggestions or comments, please do not hesitate to share them with us. We would be more than willing to engage in further discussions and make any necessary improvements.

Thank you once again for dedicating your valuable time to reviewing our work.

Dear reviewer，

Thanks for your valuable comments. As the deadline for revision (closes on November 27th) is fast approaching, we would be grateful if you could allocate some time to review our revision.

We understand that you have a multitude of responsibilities, To facilitate a swift evaluation of our revisions, we have summarized the corresponding changes as follows:

• We add the model computation complexity and model size comparison among CirT, GraphCast, and PanguWeather (Table 4 in the revision).
• We add an ablation study to compare the results of autoregressive prediction and direct prediction (Table 5 in the revision).
• We add motivation for employing the Fourier Transform (Lines 175-177).

Please let us know if you have any additional concerns or questions. we kindly request your feedback on whether our responses have satisfactorily addressed your concerns. Thank you for your time and consideration!