Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models

Thank you for providing us with your valuable feedback. We have carefully considered your questions and would like to address them as below:

Response to Weaknesses

W1. Further explain the significance of using the averaging operation.

We appreciate your concern and would like to clarify the importance of the averaging operation. Averaging is the most important operation in FL that knowledge interaction between clients, it serves two key purposes in our framework:

• Knowledge sharing: By averaging learnable generic knowledge across devices, we prevent data silos and enable each device to benefit from the collective insights of the collaboration network, while maintaining local personalization and data privacy when sensitive data are involved.

• Trade-off between personalization and generalization: The averaging operation allows devices to learn broader patterns from others, while preserving local adapter personalization. This interplay forms a robust framework for adapting to local characteristics and leveraging global knowledge, striking a balance between personalization and generalization.

Our experiments (Table 1 below) show that LM-Weather with averaging outperforms LM-Weather-Local without it, highlighting the importance of averaging operation. We hope these explanations and additional experiments can address your concerns.

Table 1. Results of LM-Weather and LM-Weather-Local (no averaging operation, i.e., no communication only updating the respective models locally at each client) on forecasting, $\downarrow$ is the average performance degradation relative to LM-Weather.

W2. Recommended to rewrite Section 3.1-4 with increased clarity and accuracy, simplifying the content for better comprehension.

We appreciate your careful review and constructive suggestions. We will thoroughly revise these content to streamline the presentation of symbol and equations and simplify the content for better clarity and readability.

Response to Questions

Q. This paper uses GPT-2 as the PLM, which is a very basic pre-trained language model. I am curious why more advanced PLMs like GPT-4, Llama, and Vicuna were not considered for better sequence modeling performance.

We acknowledge the potential benefits of using more advanced PLM. However, Our primary goal is to establish a general framework for on-device weather sequence modeling via tuning PLMs. We chose GPT-2 for stability and cost-effectiveness, allowing for a basic yet effective demonstration of our approach. While more advanced PLMs may offer better performance, they require significant computational and storage resources, which can be a challenge for deployment on low-resource weather devices. To explore the LM-Weather performance with more advanced PLMs, we conducted additional evaluations using Llama-3B and Vicuna-7B, which show that more advanced PLMs can indeed improve performance (Tables 2-5 below). Although GPT-4 was not included due to the unavailability of its weights (we are fine-tuning based on weights), our results demonstrate the potential benefits of using more advanced PLMs. Nevertheless, the increased computational demands of these models must be carefully considered for practical deployment. We hope these explanations and additional experiments can address your concerns.

The following results report the average forecasting (period [96, 192, 336, 720]) and imputation (50%) performance.

Table 2. Performance of LM-Weather with different PLM backbone (ODW1T).

Table 3. Performance of LM-Weather with different PLM backbone (ODW1V).

Table 4. Performance of LM-Weather with different PLM backbone (ODW2T).

Table 5. Performance of LM-Weather with different PLM backbone (ODW2V).

We greatly appreciate your insightful comments. We've carefully considered and addressed your concerns as follows (additional results are in Global Rebuttal PDF file):

Response to Weaknesses

W1. Some confusion about Figure 1.

We will revise Fig. 1 to clarify the explanation of panel C and unify labels to improve overall clarity and readability, and correct the typo referencing Fig. 1F to Fig. 1a.

W2. Effectiveness of the method in spatio-temporal modeling.

To further demonstrate the effectiveness of LM-Weather for spatio-temporal modeling, we conducted additional experiments with a variant, LM-Weather-ST, which aggregates site data for centralized modeling. We compared its performance to the original LM-Weather and classical spatio-temporal forecasting baselines (STGCN, ASTGCN, TCGN, and CLCRN). The results (Table 3, Global Author Rebuttal PDF file) show that LM-Weather-ST achieves optimal performance, while the original LM-Weather still significantly outperforms the baselines despite being trained in a decentralized manner. This suggests that LM-Weather has strong potential for spatio-temporal modeling tasks. Future work will explore incorporating server-side graphs to model relationships between devices, further enhancing LM-Weather's benefits for spatio-temporal modeling.

W3. Timeliness and stability of forecasts

• Timeliness: We provide inference times for various forecasting periods (Table 4, Global Author Rebuttal PDF file). When combined with the communication time of 0.29 s, these results demonstrate that LM-Weather achieves efficient prediction times, ensuring timely delivery of forecasts.
• Stability: All experiment have been conducted five times. We present the standard deviations of LM-Weather and the runner-up model (Table 5, Global Author Rebuttal PDF file), which indicate that LM-Weather can maintain stable performance over multiple experiments. Furthermore, we assessed longer-term forecasting tasks (1080, 1440, 1800, and 2160 hours) with a fixed input length of 192 (Table 6, Global Author Rebuttal PDF file). LM-Weather not only achieves optimal performance but also exhibits relatively stable error variations compared to representative baselines, further validating its stability.

W4. It is recommended that statistical charts be added.

We will add visualizations to each of the core experiment results in the revision.

Response to Questions

Q1. Why region transfer performance on ODW1 is not as good as that of GPT4.

LM-Weather outperforms FL-GPT4TS in most zero-shot scenarios, but the specific case of OWD1T -> OWD2V highlights an interesting trade-off. The difference lies in the distinct local tuning and global communication strategies employed by each method. LM-Weather uses learnable low-rank parameters to balance generic sequence modeling with personalized regional insights, whereas FL-GPT4TS unfreezes LayerNorm layers for more direct adaptation of the PLM's knowledge. This allows FL-GPT4TS to update and communicate more parameters, potentially enabling greater knowledge sharing in modeling heterogeneous weather data, but at the cost of increased communication overhead. Additionally, significant non-iid between OWD1T and OWD2V contribute to this difference.

Q2 How the low rank matrix of Theorem 4.2 ensures privacy.

We provide additional description and proofs for this, which can be found in G1, Global Author Rebuttal.

Q3. Can interpolation based on sparse stations be used to extend the model to spatial dimension prediction and improve generalizability?

While spatial interpolation on sparse stations can increase the scale of training data, it is not an effective approach for extending the model's forecasting range and enhancing its generalizability due to:

• Spatial interpolation of site data does not ensure the accuracy of the interpolated values, particularly when dealing with very sparse datasets like ODW2.
• Even when interpolating between geographically proximate stations, the biases introduced by geographical differences cannot be fully compensated by interpolation alone, especially when multiple variables are involved. The complex interactions between variables and the influence of regional meteorological features render spatial interpolation ineffective for sparse station data.

To further validate our statement, we conducted experiments on OWD1T and OWD2T using three spatial interpolation algorithms (Kriging, IDW, Spline). The results (Tables 7&8, Global Author Rebuttal PDF file) show that spatial interpolation performs poorly in sparse stations and fails to improve performance.

Q4. Stability and maximum forecasting time.

Regarding stability, please refer to our response for W3. As for the maximum forecasting time, while it is theoretically possible to extend the forecast horizon indefinitely by increasing computational resources, when the input data length remains fixed, longer forecast periods introduce greater uncertainty, leading to decreased performance. As discussed in W3, we observed a significant decline in performance when extending the forecast period from 720 to 2160 hours. This degradation highlights that, although technically feasible, very long forecasting times may not be practically useful due to the increased uncertainty and reduced accuracy.

Q5. Results when parameters are the same in the ablation experiment.

While it's true that increasing trainable parameters often boosts performance, LM-Weather-G/H's 4.5% gain comes at a steep cost: a five-fold increase in parameters. This trade-off is suboptimal. Our goal is to strike a balance between performance and efficiency. Notably, when parameter of the experiment group are similar, the original LM-Weather consistently outperforms its variants (Table 9, Global Author Rebuttal PDF File), demonstrating its optimal trade-off between performance and efficiency.

Thank you for providing us with your valuable feedback. We have carefully considered your questions and would like to address them as below:

Response to Weaknesses

W1. Justification for using PLM and clarity on weather dataset.

Although our dataset consists solely of time series data without any textual content, using a PLM as the basis of the weather foundation model offers substantial flexibility and an optimal trade-off between performance and costs. Our motivations can be summarized by:

• Limitations of Existing Climate Foundation Models: Current climate foundation models, such as ClimaX [1] (73B parameters) and Pangu-Weather [2] (up to 100B parameters), demand vast computational resources and extensive reanalysis data volumes (over 100 TB) for pre-training, making them unsuitable for deployment on low-resource meteorological equipment due to prohibitive operational costs and computational demands.
• Sequence Modeling Capabilities of PLMs: Despite being initially trained on text, their advanced sequence modeling capabilities, proven on time series analysis [3]. This eliminates the need for training models from scratch with high-quality weather observational data, which is often scarce and expensive to acquire as opposed to reanalysis data. Consequently, adopting a PLM not only employs its powerful sequence modeling abilities but also drastically cuts down on computational expenses and deployment costs when fine-tuning for weather sequence modeling.

W2. The paper's organization and clarity can be improved, e.g., notation, figure, and placement of technical details.

Thank you for your careful review and constructive comments. In our revision, we will recheck all notations in the paper to avoid confusion. Additionally, we will adjust the layout of the main text to incorporate important content from the appendix, enhancing readability. Your feedback will significantly improve the quality of our paper!

W3. The last two items in contribution part appear to be advantages rather than contributions.

We will adjust the contribution part to improve clarity of expression.

Response to Questions

Q1: Does the proposed method only focus on time series data?

Yes, LM-Weather currently specializes in processing weather time series. Utilizing a PLM can bring significant benefits in terms of flexibility and cost-effectiveness, as detailed in our response for W1. To further enhance the generality and applicability of LM-Weather, we plan to expand our dataset to include multi-modal data such as descriptive weather texts, satellite cloud imagery, and radar data, ultimately developing a comprehensive multi-modal weather sequence analysis framework.

Q2. Highlight the main research question.

The main research question can be summarized as: How can we adapt pre-trained language models to efficiently and effectively model weather sequences on local devices in a decentralized manner, while addressing the challenges of data diversity and limited computing resources? This question seeks to explore:

• Adaptation: How can complex models originally designed for language processing be tailored to understand weather sequences?
• Personalization: Can these models provide tailored weather predictions that suit the specific conditions and data available at individual locations and devices?
• On-device efficiency: Can this approach operate effectively on local devices, ensuring privacy and minimizing resource use despite the constraints of limited computational power and storage?

Q3. Federated aggregation mechanism and pseudo code.

Our aggregation mechanism is based on FedAvg, where only low-rank parameters are shared and averaged, weighted by each client's sample size ratio. We provide the pseudo code below, which will be added to the revision for clarity.

1. Receiving low-rank parameters from clients
2. Calculate the total number of samples across clients $N = \sum_{i=1}^k n_i$
3. Aggregating low-rank parameters: $\theta_{l}' = \sum_{i=1}^k \frac{n_i}{N} \theta_{l, i}$
4. Broadcasting updated $\theta_{l}'$ to clients to continue next communication round.

Q4. Explain the token embedding? Is the token to be described as a special pattern of a subsequence of the time series? How to define the vocabulary of the tokens?

The token embedding module aims to transform subsequences of weather time series into a higher-dimensional space using a convolutional layer. Each transformed subsequence is represented as a continuous vector, known as a token. This process allows the model to capture localized patterns within a specific receptive field. Each token encapsulates patterns from a subsequence of the time series, capturing local dynamics. The kernel size determines the subsequence length, combining each point with its immediate neighbors to form a token. These tokens capture important temporal dependencies. Unlike traditional NLP, the vocabulary here is formed by the diverse outputs of the convolutional layer, representing a range of patterns in the input data. This vocabulary is continuously defined, shaped by the convolutional filters that learn to identify and abstract temporal features during training.

Reference:

1. ClimaX: A foundation model for weather and climate. arXiv 2023.
2. Accurate medium-range global weather forecasting with 3D neural networks. Nature 2023.
3. "Large language models for forecasting and anomaly detection: A systematic literature review. arXiv 2024.

Thank you for your confirmation. We are pleased that our rebuttals have addressed your concerns and appreciate your continued support for our paper.

Best regards,

Authors

Thank you for providing us with your valuable feedback. We have carefully considered your questions and would like to address them as below (additional results are in Global Rebuttal PDF file):

Response to Weaknesses and Question (& means joint response)

W1. Details about the transmission strategy.

Our framework employs a synchronous federated learning strategy, where the client and server communicate at a fixed frequency (every 20 iterations). The process of transmission includes:

• 1. client-side updating: each participant uploads low-rank parameters of their local model.
• 2. server-side aggregation: the server aggregates uploaded parameters via federated averaging.
• 3. low-rank parameter broadcasting: the server broadcasts aggregated parameters to each participant for the next round.

By communicating only low-rank parameters (0.38M parameters), we minimize bandwidth consumption. We acknowledge potential issues in practical applications, including communication overhead, latency, and data privacy. Our method addresses these concerns by transmitting only low-rank parameters, thereby safeguarding privacy and minimizing communication overhead. We will clarify this key aspect of the transmission strategy in the revision to enhance understanding.

W2 & Q3. Detailed information of the FFN in parameter adapter.

Our parameter adapter is a straightforward FFN with a single linear layer, converting the PLM's output (input channels, e.g., 768 for GPT2, 4096 for Llama-3B) to align prediction horizons of [96, 192, 336, 720] (output channels). We chose this configuration to optimize the balance between performance and computational efficiency. Our additional experiments (Table 1, Global Author Rebuttal PDF file) show that adding more layers only marginally improves performance (0.21% with two or three layers) and even decreases with four layers, while significantly increasing computational demands. Therefore, the single-layer setup is the most cost-effective, providing an optimal trade-off between efficiency and performance.

W3 & Q1. Introduction and description of backgournd, and more detailed proofs and theoretical validation for theorems.

We provide additional description and proofs for theorems, which can be found in G1, Global Author Rebuttal.

W4. Important details and results should be better integrated into the main body.

Thank you for your constructive feedback. We will revise the layout to highlight the core results in the main body.

Q2. Potential confusion in the notations $n_i$ and $n_k$.

We apologize for the confusion caused by our typo. The correct Eq. 2 is: $n_i$, $F(\theta) := argmin \sum_{i=1}^{N} \frac{n_i}{n} F_i(\theta_i; \lbrace D_i \rbrace)$, where $n_i$ represents the $i$-th client. We will carefully revise and clarify these notations to enhance the readability.

Q4. Does the dataset have detailed information on extreme climate regions, and if not, what are the plans for expanding the dataset?

Including more details about specific extreme climatic regions indeed benefits the expansion of our dataset's potential application scenarios. Current version comprises real observations from various cities, with limited access to detailed topographic or regional information. Recognizing the potential benefits of expanding this coverage, we are actively working to include a broader range of climatic conditions. Specifically, we plan to introduce multimodal weather data, such as radar echoes and satellite cloud images, into our dataset's corresponding regions. Additionally, by collaborating with climate experts, we will provide textual descriptions of weather trends to create a comprehensive multimodal dataset.

Response to Limitations

L1: Future work: expanding the dataset, deepening theoretical and empirical validation, and exploring the applications and limitations of few/shot-learning.

Thank you for your insightful comments, which have provided a clear direction for our future work. As the response for Q4, we plan to introduce additional data modalities to enhance the comprehensiveness and applicability of current datasets. In future work, we will conduct more detailed theoretical and empirical analyses based on the newly constructed multimodal dataset. Additionally, we will further explore the limitations of these observations in practical applications and few/zero-shot scenarios.

L2: It would be much better integrate the essential experimental details and results into the main body.

We will integrate essential details and results into the main body in the revision to enhance readability and adjust the layout accordingly.

L3: Whether there will be a powerful foundation model for time series prediction. If the assumption is not true, the application of the proposed solution is limited.

Our approach is indeed based on the assumption that there is already a good foundation model. We believe that this assumption is reasonable because there are already many well-established foundation models that can be used in time series forecasting tasks, e.g., MOMENT[1], Timer[2]. However, we also understand the your concern that without a powerful foundation model, our method may suffer. To address this concern, we used several PLMs that weak than GPT-2 (e.g., Bert [3], CTRL [4]) to evaluate the performance of LM-Weather, and the results (Table 2, Global Author Rebuttal PDF File) show that LM-Weather can achieve good performance when the assumption of a powerful foundation model is not hold, and significantly outperforms other regular model.

Reference:

1. Moment: A family of open time-series foundation models. arXiv 2024.
2. Timer: Transformers for time series analysis at scale. arXiv 2024.
3. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv 2018.
4. Ctrl: A conditional transformer language model for controllable generation. arXiv 2019.

Thank you for your response. This approach offers an efficient solution for modeling heterogeneous and continuously updated real-world weather data, while minimizing resource demands. I believe this is a significant contribution to the field.