Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling

Dear Reviewer,

Thank you for your thoughtful review! We are pleased that you appreciate the innovative combination of AI and physics. We will address your remaining questions below.

Q1: The experimental details are not sufficient.

Thank you for your suggestions. The table below outlines the experimental details:

Q2: The time cost for your model.

As shown in the global response PDF, introducing the PDE kernel slightly increases training time, but the added computational cost is acceptable.

Q3: Error bars and p values.

The error bars of RMSE is displayed in the global response PDF.

We apply the Wilcoxon test to demonstrate that our model achieves a lower RMSE compared to the original model at the 95% confidence level (p-value<0.05).

Q4: Table 3 does not demonstrate that the proposed model is the best model. Why 120-min forecast is especially good?

Firstly, it should be pointed out that our model did not fit 30 and 90-min predictions during training, but achieving generalization in a unified model. In other words, our model achieves SOTA on most metrics under a more stringent setting (without interpolation models).

Secondly, precipitation forecasts with a lead time of 120-min or even longer serve as experimental settings [1] and play a crucial role in predicting disasters such as mudslides [2].

Thirdly, as the lead time increases, the forecasting difficulty increases, and the advantages of our model can be better demonstrated. We added the tp RMSE at 180-min to emphasize that the benefits of our model are more pronounced.

Q5: Pseudo algorithm and code.

Part of the code for the PDE kernel is included in Appendix B of the paper. The complete code will be released after the paper is accepted.

Q6: FourCastNet should have been better than ECMWF-IFS in the original paper.

In the original paper of FourCastNet [3], Figure 1 shows that RMSE z500 of FourCastNet are higher than those of ECMWF-IFS, which indicates that its prediction is relatively poor. Even in the case of FourCastNet v2 [4], its forecast skill for z500 does not surpass that of ECMWF-IFS.

Q7: All the errors in Figure 6 look alike.

The red boxes in Figure 6 show that the predictions obtained by other AI models often have problems with smoothness and lack of extreme values. On the contrary, the predictions of our model have more details and more accurate extreme values.

Q8: Explanation on router weight.

To ease your concerns, we calculate the input norm for each router as outlined in the global response PDF. The AI and physics feature norms remain consistent and similar (0.5:0.5), indicating the router's independence from both PDE kernels and attention blocks. This decoupling ensures that weight variations do not influence input. Both AI and physics branches outputs of similar magnitude, while the router dynamically selects the better part from them.

Q9: Physics is not always helping.

Using PDE kernel without changing the training method may not necessarily lead to improved RMSE. If we only use lead time=6h labels for training supervision, the relatively front PDE kernels of this deep networks will not be effectively trained due to gradient disappearance. After adding 1h and 3h supervision to the middle part of the network, the role of the PDE kernel can be better reflected.

In addition, the bias and energy metrics in the global response PDF indicate that integrating the PDE kernel effectively addresses the issue of energy decay with increasing lead time.

Q10：How do you determine the coefficient for constants.

Equation 7 is the kinematic equation in the basic atmospheric equations. $F_h$ in Equation 7 represents the friction force, which has a very small dimension (10e-12), so it can be omitted in the calculation refer to the Section 2.4 of [5]. $fv$ in Equation 15 represents the Coriolis force, where $f$ is the geostrophic parameter setting as a constant refer to the Section 4.6.3 of [5]. Other constants are also referenced from [5].

Q11: Pressure level and geopotential height are not the same thing.

While we appreciate the reminder, we would like to clarify that we do not confuse the p-coordinate and the z-coordinate. Our PDEs are all converted to the p-coordinate refer to Sections 1.6.1-1.6.2 of [5]. We will give a more detailed proof in later versions.

References

• [1] Sønderby C K, Espeholt L, Heek J, et al. Metnet: A neural weather model for precipitation forecasting.
• [2] Brunetti M T, Melillo M, Peruccacci S, et al. How far are we from the use of satellite rainfall products in landslide forecasting?
• [3] Kurth T, Subramanian S, Harrington P, et al. Fourcastnet: Accelerating global high-resolution weather forecasting using adaptive fourier neural operators.
• [4] Bonev B, Kurth T, Hundt C, et al. Spherical fourier neural operators: Learning stable dynamics on the sphere.
• [5] Holton J R. An Introduction to Dynamic Meteorology. Forth edition.

If our responses have clarified your inquiries, we kindly ask for an update to your score. Thank you very much for your time.

Dear Reviewer,

Thank you for your insightful review and detailed feedback! We are glad for your recognition of the significance of our hybrid modeling of physics and AI.

We appreciate your feedback on specific claims in our paper, and will refine specific statements and add citations of relevant papers as necessary. In the following, we will address your remaining questions.

Q1: Comparisons with other hybrid models that combine machine learning with physics.

While there exist various methodologies for integrating AI and physics, our approach diverges in both methodology and focus from those mentioned. The referenced paper primarily integrates machine learning techniques into a complete dynamical model (e.g., AGCM) to improve the forecast performance of this physics model. But our method focuses on combining PDE processes with neural networks, rather than improving the existing physical model. In addition, our focus is on finer-scale physical modeling to generalize finer-scale predictions without valid training labels, which differs from focusing on improving forecasting skills.

Q2: The time-embedding and multi-lead times in one model have also been demonstrated before, see Stormer and MetNet.

Thank you for your reminder. It should be noted that time-embedding and multi-lead times are not our core innovations (our core innovation lies in the physical modeling). They represent technical innovations in model implementation. We will increase the citations of the papers related to these concepts.

However, we would like to emphasize that these methods differ from previous work. In Stormer and MetNet, there is no direct correlation between the number of network layers and the forecast lead time. In MetNet, the lead time condition serves as input for all network layers, while our network module (HybirdBlock) evolves within a short timeframe without lead time conditions. In our approach, the lead time condition is solely fed into the decoder to extract predictions from different network layers' outputs.

For multi-lead time training, the method of our paper is a completely different technology from the Multi-step finetuning in Stormer. Stormer's finetuning focuses on autoregressive prediction, and its network itself can only output predictions for the next single step. In contrast, our method has the capability to generate predictions for multiple steps within a single forward process.

Q3: How FourCastNet, ClimODE, or Keisler was used for nowcasting?

For the precipitation nowcasting experiment, we use 1-hour interval ERA5 to train FourCastNet, ClimODE, and Keisler, respectively, to obtain models that can make 1-hour predictions. (NOTE: In the medium-range forecast experiment, these models have a lead time of 6 hours during training.) However, as ERA5 lacks half-hour data, these models are unable to directly provide half-hour predictions. Therefore, we utilize additional interpolation models to interpolate the 1-hour predictions of these models to 30-min.

Q4: In section 4.3, how are the forecasts including WeatherGFT initialized? Do they all use ERA5?

The initial fields of all models are from ERA5. The only difference is that other models need to interpolate their prediction results to 30-min, while our method can directly get 30-min of prediction from the networks.

Q5: Does this inclusion of a physics kernel improve bias? What about the conservation of energy?

This is a good suggestion and helps us to have a more comprehensive understanding of the role of the PDE kernel. We measured bias and energy and found that the PDE kernel plays a positive role in maintaining energy, as shown in global response PDF. We believe that the preservation of energy is intimately linked to enhanced physical modeling, exemplified by the dynamic equations outlined in Equation 7. We present the calculation methods and related references of bias and energy in this PDF.

Q6: What is the computational cost of having a PDE kernel compared to just the transformer by itself?

As shown in the global response PDF, introducing the PDE kernel slightly increases training time, but the added computational cost is acceptable.

Q7: Claim “In addition, the prediction error of our model at the lead time of 6-hour is significantly smaller than that of the physical dynamic model ECMWF-IFS”. For Z500 this is not supported by Figure 4.

We apply the Wilcoxon test to demonstrate that our model achieves a lower RMSE compared to the ECMWF-IFS at the 95% confidence level (p-value<0.05). The table below indicates that the improvement in z500 is relatively smaller than that in other variables when compared to the ECMWF-IFS, which results in the z500 curve in Figure 4 closely resembling that of the ECMWF-IFS.

Q8: For Figure 5 seems to be a function of the contouring in Matplotlib as the difference plots show similar magnitude of errors.

The color bar we employ is consistent and standardized, with its upper and lower bounds derived from the maximum prediction error across all forecasts. By examining the error visualization in Figure 5, it is evident that our predictions exhibit reduced errors. The experiment depicted in Figure 4 quantitatively demonstrates that our model boasts a relatively lower RMSE. Moreover, as detailed in the response to Q7, the results displayed in Figure 4 hold significance.

We will introduce additional visualizations in the appendix to illustrate that our model yields comparatively smaller prediction errors.

If our responses have clarified your inquiries, we kindly ask for an update to your score. Thank you very much for your time.

Dear Reviewer,

Thank you for your thoughtful review and detailed feedback! We are delighted that you value the innovative fusion of AI and physics in our research. We have revised the paper according to your suggestions and will now respond to the remaining queries you have.

Q1: The paper is well-structured but could benefit from clearer analyses.

Thank you for your affirmation. We will add more quantitative and rigorous analysis to the paper to fully demonstrate our model. In the global response PDF, we added the evaluation of two indicators, bias and energy. The results show that using PDE kernel can assist in making the model's prediction field energy more consistent.

Q2: The physics model relies on a limited set of PDEs.

We agree that exploring more PDEs is valuable. Currently, we have employed only a few PDEs which results in performance enhancements and demonstrates the effectiveness of our design in combining AI and PDEs. Moving forward, we intend to augment the number of PDEs to simulate atmospheric dynamics more comprehensively.

Q3: The baseline comparisons are limited.

We have included additional model comparisons, as depicted in the table below. Specifically, SphericalCNN is a CNN model designed for spherical data, DMNWP utilizes a diffusion model for weather prediction, and EWMoE employs a mixture of experts (MoE) for weather forecasting.

Q4: Can you further explain Equation 5 and $K_x$ and $M_x$?

Equation 5 shows the differential and integral operators in the model. $K_x$ is the convolution kernel. Assume a one-dimensional data $x=[-2,-1,0,1,2]$. It gradually increases from left to right by 1, that is, its gradient is 1. Applying convolution kernel $K_x$ to $x$, the result is: $Conv(x, K_x)=\frac{(-2)\times 1+(-1)\times (-8)+0\times 0+1\times 8+2\times (-1)}{12} = 1$ By using this convolution kernel, the data gradient can be determined.

$M_x$ obtains the integral through matrix multiplication. Given the matrix $x$ below, the result of $xM_x$ is:

1 & 4\\\\ 2 & 5\\\\ 3 & 6 \end{bmatrix},\ xM_x= \begin{bmatrix} 1 & 4\\\\ 2 & 5\\\\ 3 & 6 \end{bmatrix} \begin{bmatrix} 1 & 1\\\\ 0 & 1 \end{bmatrix}=\begin{bmatrix} 1 & 1+4\\\\ 2 & 2+5\\\\ 3 & 3+6 \end{bmatrix}$$ > Q5: How does the convolution layer work to align features? The size of the tensor of the output of the PDE kernel is different from that of the output of the attention. The size of the output of the PDE kernel is $[8,5,13,32,64]$, where each dimension represents *batch size, (z, q, u, v, t), pressure levels, H-patch, W-patch*. The size of the output of the attention is $[8,1024,32,64]$, where each dimension represents *batch size, embedding dim, H-patch, W-patch*. We first reshape the output of the PDE kernel to $[8,5\times 13,32,64]$. Then, through a convolution layer `Conv(in_channel=65, out_channel=1024)`, the size of the tensor is converted to $[8,1024,32,64]$, aligning with the output from the attention block. > Q6: How do you update the learnable factor $r$ and compute the weight to draw Figure 1? Below is the PyTorch code for initializing and using the learnable factor $r$. ```python import torch.nn as nn r = nn.Parameter(torch.zeros(1, 1, 1, dim), requires_grad=True) def router(physics_features, ai_features): # Features size: [B, H, W, dim] r1 = 0.5*torch.ones_like(physics_features) + r r2 = 0.5*torch.ones_like(ai_features) - r mixed_features = r1*physics_features + r2*ai_features return mixed_features ``` Through the automatic gradient function of PyTorch, $r$ will be updated through model backward. After the model training is completed, we first average the $r$ of each HybridBlock to obtain 24 values corresponding to 24 blocks. Then we divide them into 4 groups, namely: ```python # 1, 2, 3, 4, 5, ..., 24 # 00:15, 00:30, 00:45, 01:00, 01:15, ..., 06:00 [1, 5, 9, 13, 17, 21] # 15min [2, 6, 10, 14, 18, 22] # 30min [3, 7, 11, 15, 19, 23] # 45min [4, 8, 12, 16, 20, 24] # 60min ``` Then, we can average the $r$ of each group and draw `Figure 1`. > Q7: Does tp have a related PDE? How do you deal with variables that are not related to any PDE? There is no PDE specifically for precipitation, since precipitation is considered as a diagnostic variable in the atmospheric dynamics, meaning that precipitation can be derived from other atmospheric variables. For these diagnostic variables, we utilized neural networks to predict their values. While these variables may not directly influence the PDE kernel, their information will be amalgamated through the router to accomplish implicit modeling. > Q8: How fine-grained can the model achieve? Doing 30-minute forecasts with hourly data is impressive, but can the model achieve finer resolutions like 15 minutes? As given in `Figure 2` and `Section 3.6`, each PDE kernel simulates atmospheric dynamics within 300 seconds, allowing our model to make predictions at various time intervals by stacking these kernels. Theoretically, the smallest time scale for our model to forecast is 15 minutes, this is because one attention block is composed of 3 PDE kernels, equating to 3x300s=15min. Presently, we are unable to evaluate forecasts at finer scales than 30min due to the lack of ground truth at the 15min. Nevertheless, the generalization of 30min predictions showcased in the paper has already proven the efficacy of our physical-hybrid modeling approach. If our responses have clarified your inquiries, we kindly ask for an update to your score. Thank you very much for your time and feedback.

First of all, we greatly value your attention to our work and appreciate you pointing out the relevant research progress. At the stage of our research (up to April 2024), we did notice some work on physical-AI hybrid modeling, including NeuralGCM (published on ArXiv in November 2023) and ClimODE (published on OpenReview in January 2024). However, at the time of our submission, only ClimODE was fully open-source and allowed for free resolution adjustments, enabling reproducibility and comparison under the same experimental settings. Therefore, we used it as a benchmark for low-resolution weather forecasting tasks.

Regarding the NeuralGCM you mentioned, we cited this work in our paper, but prior to our submission, the GitHub repository for this model did not provide complete training code, making it difficult for us to train the model and compare under the same experimental settings.

The core innovation of our paper lies in enhancing the generalization capability of AI models in fine-grained predictions by introducing physical processes (PDE kernels). As stated in the paper, our model demonstrates good performance in multi-temporal scale generalization, especially in situations with limited training samples, such as 30-min predictions.

Moreover, as a work primarily focused on methodological innovation, we only conducted our validations on low spatial resolution (1.4 degrees) data to demonstrate the effectiveness of our method, without directly optimizing performance on high-resolution (0.25 degrees) data (for example, using relatively complex fine-tuning strategies to reduce multi-step prediction errors).

Thank you for your important points; we will take them seriously and look forward to reflecting the advancements in this field more comprehensively in our future research. We hope our response enhances the understanding and recognition of our work.