Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting

We show great gratitude to you for approving of the quality of our paper. We have make the detailed response to thoroughly release your concerns.

W1 & Q1 & Q2: Concerns on the applicability in real world.

We are sorry for not covering the deployment of Time-FFM in the manuscript. Since we make Time-FFM a "domain" level time series forecaster, we deem that federated training participants are more likely to be edge clouds with abundant computing resources, instead of resource-limited edge devices. Upon training, Time-FFM can also be deployed at edge clouds for forecasting tasks. Hence, we think that the resource limitation might not be a bottleneck of the deployment of Time-FFM and it realistic to deploy Time-FFM in real-world FL scenarios.

W2: TimesFM, as a centralized foundation model, conflicts with your motivation.

Compared with the modality of text, time series is more domain-specific and (commercial) copyright-sevsitive, i.e., private knowledge may be inferred from historical time series readings, especilly in finance and healthcare domain. Hence, it is of great significance to take data privacy into consideration in the construction of time series foundation models. Moreover, a multitude of public data cannot even be adopted for pre-training foundation models due to data license restriction. Some foundation models like MOIRAI, have released large-scale public time series data merely for research instead of commercial use.

Hence, our work uniquely bridges the gap between foundation models and federated learning, which not only enhances the privacy and applicability of foundation models in sensitive domains but also opens up new avenues for leveraging rich, yet previously inaccessible, time series data for advanced predictive analytics, addressing a crucial need in the (commercial) field.

W3&W5: More details for how to tackle data heterogeneity across clients.

We aim to learn the underlying commonalities present in time sereis data, though corss-domain time series data are of great heterogeneity. In Federated Averaging framework, all domains learn a single shared model which would perform poorly on each specific domain. If we simply train dedicated model for each domain using their own local data (even if it is small) to train a local model, the domain-specific models may not generalize well to unseen domains. Therefore, we need to strike a balance between the generalization and domain-specific prediction. We devise a persoanlized federated strategy with a global encoder and personalized heads. The global shared encoder can learn the** common temporal embeddings or representations across domains**, which is in line with the success of centralized deep learning, i.e., the sharing of global feature representations among data [1][2]. The personalized heads enables domain-specific decoders, accounting for yielding prediction results fitting for local distributions.

[1] LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 2015. [2] Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE TPAMI, 2013.

W4: It would be better to improve and clarify the technique depth.

Thanks for your insightful concerns on our contributions. We aim to propose a Language model-empowered federated foundation model for time series forecasting.

• Given the differentiation of dimensionality and horizon, we introduce the modality alignment module encomposing the channel-independent and patching techniques, which may follow the track of GPT4TS, Time-LLM, MOIRAI, etc.
• For bootstrapping the pre-trained GPT2 backbone for cross-domain time series reasoning, we propose to adaptively construct prompts from how to understand patch tokens, rather than from rigid domain instructions like Time-LLM and UniTime.
• Due to cross-domain time series heterogeneity, we devise a personalized federated strategy (different from Federated Averaging which aims at learning a global prediction model), with global encoder and personalized prediction heads.

In conclusion, we propose the first federated foundation model for time series forecasting, with adaptively generating domain-specific prompts and tackling time series heterogeneity for general-purpose learning and personzalied prediction.

W6: It would be better to convert TimesNet, DLiner, and PatchTST, into their federated version.

Thanks for your valuable suggeations. We report the performance comparison in Table 3 of the PDF in the "Author Rebuttal". In the modified version, we will supplement the experiment results and the prediction case study to make the effectiveness of Time-FFM more convincing.

Q3: The rationality of the generated prompts.

We appreciate your attention on the rationality of prompts. We want to adaptively select the "optimal" text prototypes as prompts according to the domain-sepecific patch tokens by prompt adaption module. We think the prompts indicate how the LM backbone understand the domain temporal pattterns (conveyed by patch tokens) and thus can bootstrap domain-sepecific time series reasoning, though the prompts may have conflict with how we humans understand the patch segment [1].

We evaluate the effectiveness of prompts in Table 5. We have the key observation that the prompts can improve the forecasting performance (by comparing A.1 and A.2); the proposed adaptive prompt outperforms the domain instruction (by comparing A.1 and A.3). Furthermore, the showcase in Fig. 3 illustrates that different domain datasets correspond different text prototypes (say prompts).

Hence, we think ablation results and showcase can both reflect the rationality of prompt adaption. In the modified version, we will follow your suggestion to provide more case studies and theoretical analysis to make it more convincing.

[1] Sun, C., Li, H., Li, Y., & Hong, S. TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series. In ICLR 2024.

We are happy that our responses have effectively addressed your concerns. We would like to express our gratitude again for taking the time to review our paper and provide us with such detailed and invaluable suggestions.

C1. If we consider the clients as edge clouds, which are often possessed by big companies, such as Google, many kinds of time series data are supposed to be included in these edge clouds. In such situations, training an LLM-based FL method would be less important, considering that data access restriction is not a big problem.

Thanks for providing such insightful point. We will make analysis from the perspectives of the necessity of federated learning and the adoption of the LLM-based backbone.

Necessity of Federated Learning. Actually, in the big company on the scale of Google, there exists data restriction across different departments. The time series data, especially from different bussiness scenarios, can hardly be transmitted directly to a centralized cloud server for training, given the data volume and access limitation. Hence, a foundation model trained in FL paradigm may be an optimal choice. On the other hand, different (big or small) companies can be united in the community for construct a in- or cross-domain foundation model.

Adoption of LLM. We adopt the first 6 transformer layers of GPT2 as the backbone. This also follows the track of training-from-scratch foundation models, such as MOIRAI and TimesFM, where multiple transformer layers are also adopted as the backbone. In our paper, we merely initialize the backbone with the pre-trained GPT2 parameters, which can achieve comparable performance in full-tuning or fine-tuning. As a matter of fact, with the available large-scale time series data, we can train our Time-FFM from scratch without being initilized by GPT2 parameters, like Chronos.

C2. It would be better to explain the personalized federated strategy in detail for better understanding.

Thanks for your valuable suggestion. Firstly, we want to clarify that domain personalization is a necessity. The foundation model can learn the underlying commonalities in time series data, but may not guarantee the prediction performance for a specific domain, if we directly adopt the Federated Averaging framework to learn shared global model parameters for each domain. While in centralized training mode, the pre-trained model needs to be fine-tuned firstly and then adopted for specific forecasting tasks to ensure the (personalized) prediction performance.

Secondly, we analyze the rationality and present the techinical details of the designed personalized strategy. We are inspired by the success of the centralized learning, where heterogenous data are projected to representations of the shared latent space. Hence, we can learn a global encoder to enable a shared representation space over different domains and dedicated local prediction heads for generating personalized prediction results in line with local distribution.

We will modify the manuscript according to the response to W3 and C2, and supplement a case study to illustrate the rationality of personalized strategy. We sincerely hope such revision is under your consideration.

C3. It is suggested to improve the draft by adding a section (or contents) to clarify the technical depth according to the response to W4.

We will carefully follow your suggestion to clarify the technical depth in the revised version to making the novelty of the proposed Time-FFM more convincing.

We are deeply grateful for the insightful review you have provided for our manuscript. We have made the following response.

W1 & W3: Missing literature on (1) time series foundation models, (2) deep learning methods, and (3) federated learning methods.

We show great appreciation that the suggested literature is of great value to improve our manuscript. We will cite these works and report the corresponding performance comparison in the revision.

For (1): Works in (1) aim to construct a time series "corpus" and then train a time series foundation model from scratch. The performance comparison is reported in Table 1 of the attachment PDF in the "Author Rebuttal".

For (2): We compare multiple time series forecasting methods in the manuscript and feel sorry for not covering the relevant methods you metioned. We report the performance comparison in Table 2 of the attachment PDF in the "Author Rebuttal".

For (3): There have not any literature focusing on building foundation models for time series forecasting. Therefore, we merely compare the federated fine-tuning methods in TY1.

W2.1: Rationale and contributions: (1) The motivation behind the work. (2) How this is a foundation model.

For (1): We agree with you that LLMs or LLVMs are trained on public data. Compared with the modality of text, time series is more domain-specific and (commercial) copyright-sevsitive, i.e., private knowledge may be inferred from historical time series readings, especilly in finance and healthcare domain. Hence, it is of great significance to take data privacy into account when constructing time series foundation models. Moreover, a multitude of public data cannot even be adopted for pre-training foundation models due to data license restriction, such as Kaggle public datasets. Some foundation models like MOIRAI, have released large-scale public time series data merely for research instead of commercial use.

Hence, our work uniquely bridges the gap between foundation models and federated learning, which not only enhances the privacy and applicability of foundation models in sensitive domains but also opens up new avenues for leveraging rich, yet previously inaccessible, time series data for advanced predictive analytics, addressing a crucial need in the (commercial) field.

For (2): We design Time-FFM as a foundation model for time series forecasting task, like MOIRAI and Lag-Llama. Due to the absence of public time series repositories, we merely train Time-FFM on 8 benchmark datasets. Actually, Time-FFM can be trained on the built large-scale time series archive in MOIRAI or Lag-LIama and then applied for downstream forecasting tasks.

W2.2: Why use LLMs.

In our manuscript, we adopt the first 6 Transformer layers of the pre-trained GPT2 in Time-FFM, while in the foundation models training from scratch (MOIRAI, Lag-Llama, MOMENT and TimesFM), stacked Transformer layers are also adopted as the encoder backbone. We simply freeze all parameters of Transformer layers, which can achieve comparable performance compared to fine-tuning or full-tuning (observed from Table 6 in the manuscript). Actually, with the available large-scale time series data, we can train our Time-FFM from scratch without being initilized by GPT2 parameters, like Chronos.

W2.3: (1) The contributions of the work and (2) necessity of personalized heads.

For (1): We aim to propose a LM-empowered federated foundation model for time series forecasting.

• Given the differentiation of dimensionality and horizon, we introduce the modality alignment module encomposing the channel-independent and patching techniques, which may follow the track of GPT4TS, Time-LLM, MOIRAI, Moment, etc.
• For bootstrapping the pre-trained GPT2 backbone for cross-domain time series reasoning, we propose to adaptively construct prompts from how to understand patch tokens, rather than from rigid domain instructions like Time-LLM and UniTime.
• Due to cross-domain time series heterogeneity, we devise a personalized federated strategy (different from Federated Averaging which aims at learning a global prediction model), with global encoder and personalized prediction heads.

In conclusion, we propose the first federated foundation model for time series forecasting, with adaptively generating domain-specific prompts and tackling time series heterogeneity for general-purpose learning and personzalied prediction.

For (2): Our proposed persoanlized strategy is coherent to the federated training process. Each domain participant merely uploads the updated parameters of encoders, without additional modification of local optimization process. Upon finishing federated training, a global encoder and multiple heads (one for each domain) are obtained. When applying to the downstream forecasting tasks, the global encoder and in-domain heads can be frozen and directly adopted, avoiding the process of fine-tuning.

L1: Limitations on computational cost.

We agree with you that the computational cost is potentially high even we freeze all parameters of the LM backbone. In the revised version, we will follow your suggestion by supplementing such limitation.

Furthermore, since Time-FFM works as a "domain" level time series forecaster, we think that federated training participants are more likely to be edge clouds with abundant computing resources, instead of resource-limited terminal devices. Upon training, Time-FFM can also be deployed at edge clouds for forecasting tasks. Hence, we think that the resource limitation might not be a bottleneck of the deployment of Time-FFM in real world.

We will carefully incorporate your comments in the revised paper. Considering the encouraging comments from Reviewers LqRs, PDR4, and 7qPS, we believe our research findings are worth sharing with the research community. We sincerely hope that a revision is still considered.

We express our gratitude to you for providing constructive feedback on our paper. We have addressed the specific concerns as detailed below.

W1: The overall novelty is limited. Modality Alignment, Prompt Adaption and different prediction head has been explored by previous methods.

Thanks for your insightful concerns on our contributions. We aim to propose a language model-empowered federated foundation model for time series forecasting.

• Given the differentiation of dimensionality and horizon, we introduce the modality alignment module encomposing the channel-independent and patching techniques, which may follow the track of GPT4TS, Time-LLM, MOIRAI, Moment, etc.
• For bootstrapping the pre-trained GPT2 backbone for cross-domain time series reasoning, we propose to adaptively construct prompts from how to understand patch tokens, rather than from rigid domain instructions like Time-LLM and UniTime.
• Due to cross-domain time series heterogeneity, we devise a personalized federated strategy (different from Federated Averaging which aims at learning a global prediction model), with global encoder and personalized prediction heads.

In conclusion, we propose the first federated foundation model for time series forecasting, with adaptively generating domain-specific prompts and tackling time series heterogeneity for general-purpose learning and personzalied prediction.

W2: Compared with SOTA methods from TY2, the proposed TIME-FFM does not have a significant improvement.

The methods in TY2 are centralized methods which intrinsically outperform federated methods with the same model structures. However, our proposed Time-FFM even outperforms the SOTA methods in TY2, which further indicates the effectiveness of the devised prompt adaption module and the personalized federated learning strategy.

Q1: How do you obtain $E$ from $E'$?

This is accomplished through a simple linear projection. Specifcally, given $E \in \mathbb{R}^{V \times D}$, we learn a weight matrix $W \in \mathbb{R}^{V\times V'}$ to identify a small set of text prototypes in $E' \in \mathbb{R}^{V'\times D}$. We will revise the paper accordingly and include this technical detail in the revised version.

Q4: Have you tried personalized Modality Alignment and Prompt Adaption for different domains?

Thanks you for such insightful suggestions. A.6 Time-FFM-D in Table 5 denotes the distributed version of Time-FFM, i.e., personalized modules of Modality Alignment, Prompt Adaption, and Prediction Head for each domain. As depicted in Table 5, the performance of A.6 is inferior to that of A.1 Time-FFM, which further indicates that constructing a unified model across domains outperforms training dedicated prediction models for each domain.

We express our sincere thanks for the detailed and thoughtful review of our manuscript and for the encouraging appraisal of our work. We have addressed the specific concerns and respond to the constructive recommendations detailedly as follows.

W1. Algorithm Description: On page 6, the description of the algorithm seems to contain an error where Line 6 should actually be Line 5. Please carefully check the manuscript.

Thanks for your careful readings. Line 1-5 in Algorithm describes the process of global execution and the server obtains the global model parameters by Line 5. Line 6 annotates that the following lines are meant "for local training". We are sorry for confusing you and will polish the description in the revised version.

W2. Experimental Results Discrepancy: On page 6, the value of Baseline PatchTST on ETTm1 in Table 1 differs from the values reported in reference 25. You need to provide a detailed explanation of your simulation results and why these differences occur.

As is described in the footnote of Page 5, we modify PatchTST into a "unified" version as per Reference [25]. Table 1 reports the averaged evaluation results over 4 prediction windows. The MSE and MAE values of PatchTST on ETTm1 dataset are "0.971, 0.629". In Reference [25], the averaged evaluation results of PatchTST in the type of "Models Trained Across Datasets" on ETTm1 are also "0.971, 0.629". Therefore, our simulation results agree with Reference [25].

W3. Discussion on Future Directions: The article could benefit from a more in-depth discussion of the future directions and challenges for the proposed method.

We sincerely show great gratitude for your valuable suggestion.

For challenges: We aim to propose a LM-empowered federated foundation model for time series forecasting, which is non-trivial technically, considering the following aspects.

• heterogeneous inputs: Cross-domain time series data input into the foundation model are heterogeneous in terms of dimensions and historical readings, posing evident difficulty to modality alignment.
• Rigid instructions as prompts: Prompts are adopted to bootstrap LMs for time series reasoning hinging on rigid domain-specific instructions, rather than the understanding of LMs, exhibiting poor robustness for unseen domains.
• Conflicts between generalization and personalization: The ideal foundation model needs to learn the common temporal representations across domains and simultaneously enable the personalized prediction for domain-specific inputs.

For future directions: Future directions include two aspects.

• Thanks to the adopted "personalized prediction heads", future researches may focus on constructing a unified foundation model over different time series analysis tasks, such as classification, anomaly detection, and forecasting.
• Further research should explore how to train a foundation model from scratch based on the existing and potential large-scale time series repositories.

We will follow your suggestions and supplement more in-depth challenge analysis and future directions in the modified version.