WeatherGFM: Learning a Weather Generalist Foundation Model via In-context Learning

W3. Advantages of this model in multi-task

The basic Vision Transformer can also realize a variety of understanding tasks of various data sets in the form of inAput format proposed by the author, and the effect of some tasks is better than that of WeatherGFM.

• Our WeatherGFM have achieved siginificant performance improvment than specialized models on most tasks, such as the satellite forecasting, radar satellite forecasting, GOES2Radar translation and etc. It is noteworthy that we employ a single model to handle 10 tasks instead of using 10 specialised models for 10 tasks.
• In fact, different tasks have varying levels of learning difficulty. Some simple tasks are hard to improve further because they are easier to learn to a high performance level. This phenomenon can also be observed in existing works[2,3,4]. In our experiments, the GOES-IR2GOES-IR image translation task is simpler compared to other tasks. For example, the CSI index for the GOES-IR2GOES-IR task has reached around 0.99. Our model has achieved sufficiently good performance.
• The most critical advantages of our method over specialized models is its versatility and generalization ability. Specialized model data-driven models struggle to generalize to new tasks, while our model shows versatility across multiple tasks and generalization ability on unseen tasks without any fine-tuning. This is an important attribute of AI-based weather foundation models, as mentioned in existing works[1,5].

[1] Nguyen T, Brandstetter J, Kapoor A, et al. ClimaX: A foundation model for weather and climate[J]. arXiv preprint arXiv:2301.10343, 2023.

[2] Wang X, Wang W, Cao Y, et al. Images speak in images: A generalist painter for in-context visual learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 6830-6839.

[3] Liu Y, Chen X, Ma X, et al. Unifying image processing as visual prompting question answering[J]. arXiv preprint arXiv:2310.10513, 2023.

[4] Chen X, Liu Y, Pu Y, et al. Learning a low-level vision generalist via visual task prompt[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 2671-2680.

[5] Chen S, Long G, Jiang J, et al. Foundation models for weather and climate data understanding: A comprehensive survey[J]. arXiv preprint arXiv:2312.03014, 2023.

W4. Can you explain in detail which part of the module WeatherGFM shows its advantages in various tasks?

The in-context learning strategy of our WeatherGFM shows advantages in various tasks.

• Compared with existing foundation models (e.g., ClimaX), the advantages of our model lie in its versatile ability ability to learn various tasks through in-context learning. Currently, foundation models are usually pre-trained on prediction tasks, and when it comes to downstream tasks (such as predicting different variables or super-resolution), fine-tuning of the pre-trained model is required. However, our model can learn different tasks simultaneously through the in-context learning method without the need for further fine-tuning. Even our method exhibits generalization capabilities on unseen OOD tasks without any training or fine-tuning. This is an important attribute of AI-based weather foundation models, as mentioned in existing works[1,2].
• In the Appendix C section of our paper, we conducted a comparison between our model and ClimaX on the temperature at 2 meters above ground (T2m) prediction task within the ERA5 dataset (WeatherBench). ClimaX fine-tunes the pre-trained model for each lead time. In other words, ClimaX requires N models for weather forecasting at N lead times. In contrast, our universal model can handle all lead time tasks using just a single model.

[1] Nguyen T, Brandstetter J, Kapoor A, et al. ClimaX: A foundation model for weather and climate[J]. arXiv preprint arXiv:2301.10343, 2023.

[2] Chen S, Long G, Jiang J, et al. Foundation models for weather and climate data understanding: A comprehensive survey[J]. arXiv preprint arXiv:2312.03014, 2023.

W5. There are two clerical errors: (a) "the" should be capitalized. (b) The basic learning rate in line 366 should be 1e-4.

We have revised the description in our main paper, thanks to your suggestions!

Dear Reviewer qe7B,

We appreciate your thoughtful and positive feedback. We have responded to the concerns mentioned in your comments below and hope that our clarifications will further strengthen your confidence in our work.

W1. The introduction of mixed-modal masked image modeling (MMIM) pipeline.

Thank you for your suggestions. we have made detailed revisions in the main paper, please refer to page 5. The mixed-modal masked image modeling process can be formulated as follows:

$$P' _ {\text{target}}, X' _ {\text{target}} = F _ {\tau} (P _ {\text{in}}, M(P _ {\text{target}}), X _ {\text{in}}, M(X _ {\text{target}}); \theta);$$

where we randomly conduct mask operation $M$ on the prompt target $P _ {target}$ as well as the ground truth $X _ {target}$ according to the mask ratio. Meanwhile, the prompt input $P _ {in}$ and the input query $X _ {in}$ will be retained entirely. $P^{'} _ {target}$ and $X^{'} _ {target}$ represent the predicted target output of model $F _ {\tau}$. The optimization objectives are as follows:

$$\mathrm{L}^{total} _ {\theta} = \mathrm{L} _ 2(P^{'} _ {target},P _ {target}) + \mathrm{L} _ 2(T^{'} _ {target},T _ {target}).$$

where we use MSE (mean square error) loss $\mathrm{L _ 2}$ to train the weather generalist foundation model. In the inference stage, we keep the $P _ {in}$, $P _ {out}$, and $X _ {in}$ intact while the target image is fully masked. This target full masking strategy allows generalist foundation models to generate the corresponding target through a visual question-and-answer format.

W2. Evaluation Metrics

2.1 Should ACC be taken into account?

• Given that our current training and testing data are primarily based on the SEVIR dataset, which includes a large number of storm-related events, it is more appropriate to use the Threat Score/Critical Success Index (CSI) as a metric for evaluating extreme events. Previous works, such as SEVIR [1], CasCast [2], and EarthFormer [3], typically use CSI as a primary evaluation metric. The Anomaly Correlation Coefficient (ACC) is also an important metric in weather forecasting. It assesses the quality of a forecasting system by calculating the correlation between forecasts and observations. This metric is suitable for evaluating global medium-range forecast datasets, such as ERA5. In our extended experiments, we used the ERA5 dataset for testing and employed RMSE and ACC as evaluation metrics.

2.2 Can you simply explain the significance of CSI threshold setting?

• For the CSI threshold settings of radar images, we followed SEVIR [1] and chose thresholds based on the six Video Integrator and Processor (VIP) intensity levels, corresponding to pixel values [16, 74, 133, 160, 181, 219]. Similarly, for satellite images, we selected representative thresholds based on the level information provided in SEVIR dataset for different variables.

[1] Veillette M, Samsi S, Mattioli C. Sevir: A storm event imagery dataset for deep learning applications in radar and satellite meteorology[J]. Advances in Neural Information Processing Systems, 2020, 33: 22009-22019. [2] Gong J, Bai L, Ye P, et al. Cascast: Skillful high-resolution precipitation nowcasting via cascaded modelling[J]. arXiv preprint arXiv:2402.04290, 2024. [3] Gao Z, Shi X, Wang H, et al. Earthformer: Exploring space-time transformers for earth system forecasting[J]. Advances in Neural Information Processing Systems, 2022, 35: 25390-25403.

W3: Experimental results are weak.

In weather forecasting and super-resolution, the performance of the ViT baseline is nearly the same as WeatherGFM, which suggests only a minute (if any) improvement with the author’s designed modifications and pre-training strategy.

• Our WeatherGFM have achieved siginificant performance improvment than task specialized models (e.g., ViT-ST) on most tasks, such as the satellite nowcasting, radar satellite nowcasting, and GOES2Radar translation. It is noteworthy that we employ a single model to handle all tasks without any task-specific fine-tuning.
• In fact, different tasks present varying levels of learning difficulty. Some simpler tasks are challenging to improve further because they can be easily trained to achieve a relatively high level of performance. This phenomenon can also be witnessed in PromptGIP [1] and GenLV [2]. In our experiments, the satellite radar image super-resolution task is simpler compared to other tasks. For example, the CSI index for the satellite super-resolution task has reached around 0.99. Our model has achieved sufficiently good performance.

[1] Liu Y, Chen X, Ma X, et al. Unifying image processing as visual prompting question answering[J]. arXiv preprint arXiv:2310.10513, 2023.

[2] Chen X, Liu Y, Pu Y, et al. Learning a low-level vision generalist via visual task prompt[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 2671-2680.

W4. Presentation clarity.

4.1 Figure 3 is quite unclear.

We have revised Figure 3, adding corresponding descriptions for each module and formula in the training and testing process within the figure 3.

4.2 The actual pre-training method is not described extensively.

In fact, our method section does not involve any pre-training. To help you understand, we have added detailed descriptions of Mixed-modal mask modeling of Weather in-context learning in page5 of main paper.

4.3 Qualitative outputs from the model are not compared with qualitative outputs from other baselines.

In our response to Weakness 1 and 2, we provided a comparison with Climax and the quantitative evaluation on OOD tasks.

4.4 Critical details on training and inference are missing

We have added a detailed description of data construction on page 6 of the main text, and on page 15 of the supplementary materials, we have included descriptions of the network architecture and training details for the baseline UNet, ViT, and our WeatherGFM.

4.5 Are there ablations on the length of the input prompt sequence and how that affects performance?

During the design of the prompt, its length aligns with that of the model's input query and output target. More precisely, the prompt input ($P_{in}$) and the prompt target ($P_{target}$) are chosen from the training dataset based on the task types associated with the input query ($X_{in}$) and the ground truth ($X_{target}$). In other words, for a particular task, the length of the prompt remains fixed. We have performed an ablation study on the approach of selecting the prompt in section 4.4 of main paper (page. 9). The detailed information can be found in Appendix F.

4.6 Keeping text bold in the introduction paragraphs hampers readability. Line 159-180: Define what t is and what its range is.

We have revised the description in our main paper, thanks to your suggestions.

W2: Experimental comparisons are missing.

2.1 The authors don’t seem to compare WeatherGFM on the same tasks used in the Climax or Aurora papers, and it is unclear why this comparison isn’t made.

• Our approach focuses on weather understanding tasks within meteorological satellite and radar data scenarios, whereas ClimaX and Aurora concentrate on atmospheric variable scenarios based on reanalysis data. Reanalysis data scenario focuses on atmospheric variables forecesing, while meteorological satellite and radar observation scenarios involve a more diverse range of tasks, such as weather image translation, spatiotemporal super-resolution and short-term forecasting. Notably, radar short-term forecasting and atmospheric variable forecasting are very different tasks that cannot be directly compared. In fact, our method has proven the capability of generalist models in forecasting tasks for both satellite and radar data.
• To address your concerns, we have expanded our tasks to include atmospheric variables forecasting. We compared our WeatherGFM with Climax and the operational Integrated Forecasting System (IFS) on WeatherBench. Table 1 shows that our approach significantly outperforms ClimaX in the 120-hour and 168-hour forecast results, even surpassing the IFS method of ECMWF. Our method achieves comparable results to ClimaX at other lead times. It is important to note that Climax fine-tunes the pre-trained model for each lead time, necessitating N models for weather forecasting at N lead times. In contrast, our generalist model can manage all lead time tasks with a single model, eliminating the need for task-specific fine-tuning.

Note: Comparison of RMSE and ACC across different lead times for IFS, ClimaX, and ours WeatherGFM on the temperature at 2 meters above ground (T2m) variable. ClimaX views predicting at each lead time as a separate task and fine-tunes a separate model for every individual task. In contrast, our WeatherGFM utilizes a single model to deal with all of these tasks.

2.2 Moreover, only 3 qualitative examples on out-of-distribution (OOD) examples are given in Figure 6. Quantitative evaluations on OOD tasks are quite important for a foundation model.

The quatitative results have been added to Appendix H. We conduct generalization tests directly on out-of-distribution (OOD) tasks without any training or fine-tuning of our model. In contrast, the specialized models UNet and ViT are retrained for each task. Quantitative evaluation shows that our method outperforms the specialised ViT model in IR107 Satellite Extrapolation. Although there is a slight performance decrease in Tasks IR107 to IR069 and Temporal SR at 15min compared to ViT models, our WeatherGFM still significantly surpasses the results of the specialized UNet models. Our experiments demonstrate that our approach maintains generalizability in OOD tasks without requiring any fine-tuning.

Quantitative evaluation on OOD tasks

Note: * denotes that our WeatherGFM has not been trained or fine-tuned for the tasks listed below and performs generalized inference directly. # indicates that UNet and ViT undergo supervised training on the corresponding training dataset.

IR107 Satellite Extrapolation

IR107 to IR069

Temporal SR at 15min

Dear Reviewer CC3i,

Thank you for your thorough feedback. However, we feel there may have been some misinterpretations of crucial elements of our work, which could influence your assessment. We earnestly hope that our clarifications will prompt you to revisit and reassess our work.

W1: Lack of significant novelty.

1.1 While the authors do propose changes to the ViT pre-training setup used in other Weather foundation model setups, these are not significant.

We'd like to highlight the differences between the pre-trained foundation models (Climax, Aurora and SateMAE) and the generalist foundation model (our WeatherGFM). As stated in Table 1 of our paper, pre-trained foundation models (such as ClimaX, Aurora) all require fine-tuning for downstream tasks.

A pre-trained foundation model typically involves two stages: (1) pre-training with mask modeling; and (1) fine-tuning for specific tasks. For example, in SatMAE [1], the pre-trained model is fine-tuned for classification and segmentation tasks. In contrast, our approach utilizes a generalist model that eliminates the need for a fine-tuning process. Our generalist foundation model adopts the in-context learning paradigm, enabling unified training across multiple tasks simultaneously. It is important to note that the mask modeling within in-context learning is designed for visual question answering, rather than serving as pre-training for later fine-tuning. Consequently, our generalist model does not require any fine-tuning after the training phase.

Reviewer vbFm also recognized the absence of a fine-tuning requirement in our method as a significant strength. Our WeatherGFM can handle different tasks with just visual prompts, eliminating the need for additional fine-tuning. We have incorporated discussions on pre-trained foundation models, including SatMAE [1], SatMAE++ [2], and S2MAE [3], on page 3 of our main paper.

1.2 The authors’ claims of supporting multi-modality isn’t well supported considering other foundation models already do support inputs from different weather sensors and earth observation (EO) systems.

As stated in Table 1 of our paper, our WeatherGFM is the first to support single-modal, multi-modal, and temporal modalities of meteorological satellite and radar data for multiple weather undersatanding tasks. To our knowledge, no other work has achieved this. Large remote sensing foundation models (e.g., SatMAE, SatMAE++) tend to conduct image classification and segmentation tasks based on visible and hyperspectral data. Climate foundation models (e.g., ClimaX, Aurora) focus on atmospheric variables forecasting based on reanalysis data. These foundational models do not support the processing of various weather understanding tasks involving meteorological satellite and radar data.

1.3 The authors don’t provide any evidence that the techniques of Climax or Aurora wouldn’t also extend to this additional input.

ClimaX and Aurora are capable of accomplishing various downstream tasks through the approach of fine-tuning. However, they require separate models for each task, meaning that for N downstream tasks, N different models must be fine-tuned. In contrasts, our generalist model uses a single model to handle all tasks.

[1] Cong Y, Khanna S, Meng C, et al. Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery[J]. Advances in Neural Information Processing Systems, 2022, 35: 197-211.

[2] Noman M, Naseer M, Cholakkal H, et al. Rethinking transformers pre-training for multi-spectral satellite imagery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 27811-27819.

[3] Li X, Hong D, Chanussot J. S2MAE: A Spatial-Spectral Pretraining Foundation Model for Spectral Remote Sensing Data[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 24088-24097.

Dear Reviewer CC3i,

We are eager to ensure that we have adequately addressed your concerns and are prepared to offer further clarifications or address any additional questions you may have.

Should you find that our revisions have satisfactorily addressed your previous concerns, we would be most grateful if you would reconsider the evaluation of our paper with a view to enhancing its standing.

We would like to express our heartfelt gratitude for the time and effort you have dedicated to reviewing our work. It has been a pleasure to engage with you throughout this process.

Best regards,

Authors

Dear Reviewer CC3i,

We hope this message finds you well. As we approach the conclusion of the rebuttal discussion, we would be grateful if you could spare some time to go through our response. We are eager to know if we have adequately addressed your concerns, and we’d be more than happy to provide further clarifications or answer any additional questions you may have.

Once again, thank you for your time and dedication to this process.

Best Regards,

The Authors

After reading general comments, I would like to offer a different opinion because I am very familiar with weather forecasting.

The proposed generalist model primarily addresses many interrelated tasks within satellite and radar data scenarios, which are significantly different from medium-range forecasting tasks. In my experience, medium-range forecasting tasks involve hundreds of variable levels, with a complexity of optimization that far exceeds weather understanding tasks from satellite and radar. Nonetheless, the generalist model has even outperformed ClimaX in some representative variables. This finding is deeply impressive. Therefore, in my view, the current experimental results sufficiently validate the capabilities of atmospheric forecasting.

Additionally, following ClimaX's approach and directly employing UNet and ViT to construct a baseline is reasonable. This common comparative method is often used to verify model performance across different meteorological scenarios. Thus, as the first attempt to explore a generalist model, I believe the current experiments are sufficiently comprehensive, and further investigation may be reserved for future studies. In my opinion, being too harsh is not helpful to the community.

Note: Comparison of RMSE and ACC across different lead times for IFS, ClimaX, and ours WeatherGFM on the temperature at 2 meters above ground (T2m) variable prediction task. ClimaX views predicting at each lead time as a separate task and fine-tunes a separate model for every individual task. In contrast, our WeatherGFM utilizes a single model to handle all of these tasks.

[1] Wu S, Fei H, Qu L, et al. Next-gpt: Any-to-any multimodal llm[J]. arXiv preprint arXiv:2309.05519, 2023.

[2] Lu J, Clark C, Zellers R, et al. Unified-io: A unified model for vision, language, and multi-modal tasks[C]//The Eleventh International Conference on Learning Representations. 2022.

W3: The motivation on real-world application.

3.1 There are not many combinations of tasks here.

• In practical application scenarios, regional weather forecasting often requires both weather forecasting and super-resolution tasks to achieve more accurate forecasting results. Weather monitoring scenarios necessitate a variety of weather observation data for analysis, typically requiring models to synthesize missing data, such as generating missing weather radar data, synthesizing missing shortwave infrared and visible light data at night. We can flexibly handle multiple tasks with just single model, eliminating the need to embed multiple models in practical application scenarios.
• In addition, existing work[1,2,3] has mentioned that meteorological system modeling involves a variety of different tasks that constantly interact in intricate ways.

3.2 There is no obvious improvement over the conventional specialist model.

• Our WeatherGFM have achieved siginificant performance improvment than specialized models on most tasks, such as the satellite nowcasting, radar satellite nowcasting, GOES2Radar translation and etc. It is noteworthy that we employ a single model to handle 10 tasks instead of using 10 specialised models for 10 tasks.
• In fact, different tasks have varying levels of learning difficulty. Some simple tasks are hard to improve further because they are easier to learn to a high performance level. This phenomenon can also be observed in PromptGIP and GenLV. In our experiments, the GOES-IR2GOES-IR image translation task is simpler compared to other tasks. For example, the CSI index for the GOES-IR2GOES-IR task has reached around 0.99. Our model has achieved sufficiently good performance.

3.3 What is the essential advance of the proposed model compared with per-task specialists?

• The most critical advancement of our method over specialized models is its versatility and generalization ability. If there are 10 tasks to be processed, specialized models require training 10 separate models while our generalist model only needs a single model to handle all 10 tasks. Specialized model data-driven models struggle to generalize to new tasks, while our model shows versatility across multiple tasks and generalization ability on unseen tasks without any fine-tuning. This is an important attribute of AI-based weather foundation models, as mentioned in existing studies[1,3].
• In the Appendix C section of our paper, we conducted a comparison between our model and ClimaX[3] on the temperature at 2 meters above ground (T2m) prediction task within the ERA5 dataset (WeatherBench). ClimaX fine-tunes the pre-trained model for each lead time. In other words, ClimaX requires N models for weather forecasting at N lead times. In contrast, our universal model can handle all lead time tasks using just a single model.

[1] Chen S, Long G, Jiang J, et al. Foundation models for weather and climate data understanding: A comprehensive survey[J]. arXiv preprint arXiv:2312.03014, 2023.

[2] Veillette M, Samsi S, Mattioli C. Sevir: A storm event imagery dataset for deep learning applications in radar and satellite meteorology[J]. Advances in Neural Information Processing Systems, 2020, 33: 22009-22019.

[3] Nguyen T, Brandstetter J, Kapoor A, et al. ClimaX: A foundation model for weather and climate[J]. arXiv preprint arXiv:2301.10343, 2023.

Dear Reviewer Broi,

Thank you for your thoughtful and positive feedback. We have addressed the concerns raised in your comments below and hope that our clarifications will further enhance your confidence in our work.

W1: The training objective in Figure 3 is unclear.

What is the motivation for designing Mixed-modal mask modeling? It seems like there should be some references that motivate this design.

Thank you for your suggestions. we have added motivation for designing Mixed-modal mask modeling and references in the main text, please refer to page 5. Inspired by the concept of Visual Question Answering [1,2,3], we introduce mixed-modality masking on various weather modalities for visual question-and-answer modeling in weather understanding tasks. This process can be formulated as follows: $P' _ {\text{target}}, X' _ {\text{target}} = F _ {\tau} (P _ {\text{in}}, M(P _ {\text{target}}), X _ {\text{in}}, M(X _ {\text{target}}); \theta);$ where we randomly conduct mask operation $M$ on the prompt target $P _ {target}$ as well as the ground truth $X _ {target}$ according to the mask ratio. Meanwhile, the prompt input $P_{in}$ and the input query $X_{in}$ will be retained entirely. $P^{'} _ {target}$ and $X^{'} _ {target}$ represent the predicted target output of model $F _ {\tau}$. The optimization objectives are as follows:

$$\mathrm{L}^{total} _ {\theta} = \mathrm{L} _ 2(P^{'} _ {target},P _ {target}) + \mathrm{L} _ 2(T^{'} _ {target},T _ {target}).$$

where we use MSE (mean square error) loss $\mathrm{L _ 2}$ to train the weather generalist foundation model. In the inference stage, we keep the $P _ {in}$, $P _ {out}$, and $X _ {in}$ intact while the target image is fully masked. This target full masking strategy allows generalist foundation models to generate the corresponding target through a visual question-and-answer format.

[1] Wang X, Wang W, Cao Y, et al. Images speak in images: A generalist painter for in-context visual learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 6830-6839. [2] Liu Y, Chen X, Ma X, et al. Unifying image processing as visual prompting question answering[J]. arXiv preprint arXiv:2310.10513, 2023. [3] Chen X, Liu Y, Pu Y, et al. Learning a low-level vision generalist via visual task prompt[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 2671-2680.

W2: The patch embedding layer is task-specific.

The patch embedding layer is task-specific. Does this mean that there is a hard code to select which patch embedding layer is used to handle different tasks or data modalities? If so, can this model be claimed as a unified model?

Our patch-embedding layer provides a degree of generalizability due to its modality-specific design. Furthermore, it can effectively accommodate new modalities.

• We currently use three types of modality-specific patch-embedding layers that can handle 10 tasks. If a new task fits within these three modalities, there's no need to add a new embedding layer. For instance, the carbon dioxide image translation task using a new dataset can directly utilize the existing embedding layer without requiring any modifications.
• If a new modality emerges, our framework can adapt by adding a new modality-specific patch-embedding layer to accommodate tasks related to this new modality. Just as in the Appendix C section of the paper, we conducted a comparison between our model and ClimaX on the task of predicting the temperature at 2 meters above ground (T2m) within the WeatherBench. In this task, the input consists of 48 ECMWF (European Centre for Medium-Range Weather Forecasts) variables, which are significantly different from the inputs of our original tasks. For this new task, we introduced a new patch embedding and adopted the approach of continuous learning to train for it, and ultimately achieved competitive results when compared to specialized models.
• Unified AI agents in the general domain, such as NeXT-GPT [1] and UNIFIED-IO [2], will also utilize different task-specific encoders for different modalities when handling inputs of various modalities. This implies that when dealing with newly introduced modalities, these models still need to adopt new encoders. The core of WeatherGFM is to construct a generalist model that can handle multiple tasks with just a single model. Meanwhile, when encountering tasks involving new modalities, WeatherGFM is scalable. As this is the first work on a weather generalist foundation model, there is potential for future advancements to enhance our model's adaptability across different modalities.

Dear Reviewer Broi,

We are eager to ensure that we have adequately addressed your concerns and are prepared to offer further clarifications or address any additional questions you may have.

Should you find that our revisions have satisfactorily addressed your previous concerns, we would be most grateful if you would reconsider the evaluation of our paper with a view to enhancing its standing.

We would like to express our heartfelt gratitude for the time and effort you have dedicated to reviewing our work. It has been a pleasure to engage with you throughout this process.

Best regards,

Authors

Dear Reviewer Broi,

We hope this message finds you well. As we are nearing the end of rebuttal discussion, we would greatly appreciate it if you could find some time in your busy schedule to review our response and let us know if you have any further questions or require additional clarification on any points.

We sincerely appreciate your valuable time and effort.

Best Regards,

The Authors

Dear Reviewer Broi,

We hope this message finds you well. We apologize for any disruption caused by contacting you over the weekend. Thank you for your initial positive rating. As the rebuttal period comes to a close, we would like to confirm whether our rebuttal has resolved your concerns and further enhanced your confidence in our work. We are more than happy to provide further clarifications or answer any additional questions you may have.

Once again, thank you for your time and dedication to this process.

Best regards,

The Authors

Dear reviewer Broi,

We sincerely appreciate your response and are pleased to know that we have largely addressed your concerns. Thank you once again for your valuable feedback, which has significantly contributed to enhancing the quality of our paper.

Best regards,

Authors

The Effect of Dataset Size

Note: ViT-ST: single-task ViT. WeatherGFM-4Tasks: our WeatherGFM trained on 4 tasks. WeatherGFM-10Tasks: our WeatherGFM trained on full (10) tasks.

Radar Temporal SR

GOES2GOES

GOES2Radar

Radar Spatial SR

W4: Weaker performance in weather image translation tasks

Why does WeatherGFM show weaker performance in weather image translation compared to other tasks?

• As demonstrated in the table below, after an additional 5 epochs of training, we observed an improvement in the performance of weather image translation (GOES-IR2GOES-Visible task). We have updated the corresponding results, which are highlighted in blue in Table 2 of the main paper.

GOES-IR2GOES-Visible

• The primary reason for the weaker performance in this task is that only the GOES-IR2GOES-Visible task uses GOES-Visible as its generation target. In contrast, when radar data is the generation target, it encompasses a greater number of tasks, such as Radar Extrapolation and GOES2Radar. As a result, the GOES-IR2GOES-Visible task is more challenging to train and requires more training epochs to achieve improved results.
• In fact, different tasks have varying levels of learning difficulty. Some simple tasks are difficult to be improved further since they can be easily trained to a relatively high performance level. This phenomenon can also be witnessed in PromptGIP [1] and GenLV [2]. In our experiments, the satellite radar image super-resolution task is simpler compared to other tasks. For example, the CSI index for the satellite super-resolution task has reached around 0.99. Our model has achieved sufficiently good performance.

[1] Liu Y, Chen X, Ma X, et al. Unifying image processing as visual prompting question answering[J]. arXiv preprint arXiv:2310.10513, 2023.

[2] Chen X, Liu Y, Pu Y, et al. Learning a low-level vision generalist via visual task prompt[C]//Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 2671-2680.

W5: A misuse of double lines for "GOES2Radar, CSI/219."

Thank you for your suggestion. We have made the revisions.

Dear Reviewer vbFm,

We appreciate your valuable feedback and would like to address the concerns you've raised. We hope the clarifications provided below will resolve your concerns and enhance your evaluation of our work.

W1: Framework's generalization ability.

It appears that WeatherGFM uses task-specific patch embedding layers, which may limit the framework's generalization ability.

Our patch-embedding layer provides a degree of generalizability due to its modality-specific design. Furthermore, it can effectively accommodate new modalities.

• We currently use three types of modality-specific patch-embedding layers that can handle 10 tasks. If a new task fits within these three modalities, there's no need to add a new embedding layer. For instance, the carbon dioxide image translation task using a new dataset can directly utilize the existing embedding layer without requiring any modifications.
• If a new modality emerges, our framework can adapt by adding a new modality-specific patch-embedding layer to accommodate tasks related to this new modality. As this is the first work on a weather generalist foundation model, there is potential for future advancements to enhance our model's adaptability across different modalities.

W2: Comparison with other climate foundation models

It would be beneficial to compare WeatherGFM’s performance against other climate foundation models (e.g., Aurora, ClimaX) using the ERA5 dataset.

To address your concerns, we have expanded our tasks to include atmospheric variables forecasting. We compared our WeatherGFM with Climax and the operational Integrated Forecasting System (IFS) on WeatherBench. Table 1 shows that our approach significantly outperforms ClimaX in the 120-hour and 168-hour forecast results, even surpassing the IFS method of ECMWF. Our method achieves comparable results to ClimaX at other lead times. It is important to note that Climax fine-tunes the pre-trained model for each lead time, necessitating N models for weather forecasting at N lead times. In contrast, our generalist model can manage all lead time tasks with a single model, eliminating the need for task-specific fine-tuning.

Note: Comparison of RMSE and ACC across different lead times for IFS, ClimaX, and ours WeatherGFM on the temperature at 2 meters above ground (T2m) variable prediction task. ClimaX views predicting at each lead time as a separate task and fine-tunes a separate model for every individual task. In contrast, our WeatherGFM utilizes a single model to handle all of these tasks.

W3: Unclear Sources of Performance Gains

It’s unclear whether the performance gains are due to dataset scalability or multi-task training or larger number of parameters.

In our paper, Figure 7 demonstrates the improvements in model scale and parameters across various tasks. As the number of tasks increases, the data size also expands. We have combined these two aspects to assess performance enhancements derived from both data and tasks.

As shown in the table below, WeatherGFM-4tasks is trained on four tasks: Radar Temporal SR, GOES2GOES, GOES2Radar, and Radar Spatial SR. It uses more data and encompasses more tasks than the ViT-ST specialized model, which is trained on these four tasks separately. However, WeatherGFM-4tasks utilizes less data and fewer tasks compared to WeatherGFM-10tasks, which is trained on ten tasks.

The results indicate that for radar image generation tasks (Radar Temporal SR, GOES2Radar, and Radar Spatial SR), both WeatherGFM-4tasks and WeatherGFM-10tasks outperform ViT-ST. This is likely because most of the selected tasks are related to radar image generation. However, for the satellite image generation task (GOES2GOES), WeatherGFM-4tasks does not perform as well. This suggests that multi-task learning of similar tasks can enhance the model's performance on those tasks.

Dear Reviewer vbFm,

We are eager to ensure that we have adequately addressed your concerns and are prepared to offer further clarifications or address any additional questions you may have.

Should you find that our revisions have satisfactorily addressed your previous concerns, we would be most grateful if you would reconsider the evaluation of our paper with a view to enhancing its standing.

We would like to express our heartfelt gratitude for the time and effort you have dedicated to reviewing our work. It has been a pleasure to engage with you throughout this process.

Best regards,

Authors

Dear Reviewer vbFm,

We hope this message finds you well. As we are nearing the end of rebuttal discussion, we would greatly appreciate it if you could find some time in your busy schedule to review our response and let us know if you have any further questions or require additional clarification on any points.

We sincerely appreciate your valuable time and effort.

Best Regards,

The Authors

Dear Reviewer vbFm,

We hope this message finds you well. We apologize for any inconvenience caused by reaching out over the weekend. We noticed that your initial rating of our work was slightly negative. As the rebuttal discussion period is coming to a close, we would greatly appreciate your feedback on whether our rebuttal has effectively addressed your concerns. Please let us know if you have any further questions, and we would be more than happy to assist you.

We sincerely appreciate your valuable time and effort.

Best Regards,

The Authors

Dear Reviewer vbFm,

We hope this message finds you well. We apologize for reaching out again over the weekend. As the rebuttal discussion period ends in two days, we would greatly appreciate your review of our rebuttal. We are eager to ensure we have fully addressed your concerns and would be happy to answer any further questions you may have.

We sincerely appreciate your valuable time and effort.

Best Regards,

The Authors

Dear Reviewer vbFm,

We hope this message finds you well. We are writing to gently remind you that the rebuttal discussion period will conclude in less than 30 hours. We are keen to ensure that we have thoroughly addressed all your concerns and are ready to respond to any additional questions or clarifications you might have.

Your insights are extremely valuable to us, and we greatly appreciate the time and effort you have dedicated to reviewing our work. Please let us know if there is anything further we can provide to assist in your review.

Thank you once again for your attention and support.

Best regards,

The Authors

Dear Reviewer vbFm,

We hope this message finds you well. We apologize for reaching out again, but with the rebuttal discussion period closing in the next 5 hours, we wanted to draw your attention to some key points in our response.

In particular, we would like to highlight the additional experiments conducted on the ERA5 dataset. Our model has outperformed other fundation models (e.g., ClimaX) in some representative variables. Furthermore, we have included experiments with varying data sizes, which demonstrate that multi-task learning of similar tasks can enhance the model's performance. We sincerely hope that our clarifications might encourage a reevaluation of our work.

We appreciate your understanding and your time. Please let us know if there is anything further we can provide to assist in your review.

Best regards,

The Authors