Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts

Dear reviewer, thank you for your thorough review and constructive feedback on our paper. We appreciate the time you have taken to provide valuable insights. Please find our responses below.

[C1] Time series models utilising Mixture of Experts (MoE) have been well studied since the 1990s [1] up until now [2,3,4]. As the work under review combines well-established MoE approaches with existing time series models for a specific task, i.e. forecasting, it offers a limited technical contribution to the field of time series analysis.

Thank you for the comment. We respectfully disagree with the reviewer’s perspective. In this study, the use of MoE is specifically designed to address the heterogeneity of time series during time series foundation model pretraining. This approach is novel in tackling this challenge, positioning Moirai-MoE as the first MoE-based foundation model for time series.

Regarding the reference papers you provided, [1, 2, 3] are not time series foundation models, and their motivations and experimental setups differ significantly from ours. [4] refers to Time-MoE, which is a concurrent work being submitted to ICLR 2025. It is unfair to evaluate the novelty of our paper based on a concurrent work.

[C2] The related work section is incomplete, as the authors fail to discuss existing time series model that utilise MoE. Previous works stated above [1,2,3,4] and further works should be included by the authors to provide a representative overview of the current literature.

Thank you for this helpful comment. We have revised Section 2 of the paper (highlighted in red).

[C3] The methodology section of the paper is poorly elaborated.

Thank you for this helpful comment. We have revised Section 3.2 of the paper to address this point, with the changes highlighted in red.

[C4] The authors do not elaborate on the limitations of their work.

Thank you for this helpful comment. We have added this section in Appendix C of the revised paper (highlighted in red).

[C5] The authors do not provide code to support reproducibility.

Thank you for bringing up this concern. Code is available at the link: https://anonymous.4open.science/r/moirai_moe-NB88

[C6] How does the choice of pre-trained model influence the resulting cluster centroids and, consequently, the downstream performance? To this end, have the authors investigated any other pre-trained model besides Moirai?

Thank you for this insightful comment. We have conducted additional experiments, the details of which are outlined below. Specifically, we utilized the Chronos-Small checkpoint, as it matches the number of Transformer layers in Moirai-MoE-Small. Similar to Moirai-MoE, we performed inference on Chronos using our pretraining dataset, LOTSA, to extract the hidden states after the attention modules and generate cluster centroids at each layer. Subsequently, we pretrained a Moirai-MoE-Chronos variant using the token clusters derived from Chronos. The results on Monash, presented below, show that the performance of this variant is comparable to Moirai-MoE, demonstrating that our proposed gating function is capable of leveraging knowledge from other foundation models.

Dear reviewer JYvV,

Thanks for your valuable review, which has inspired us to improve our paper further. May we know if our response addresses your main concerns? If so, we kindly ask for your reconsideration of the score. We look forward to hearing your thoughts on our revisions.

Thank you very much for the response and the revised manuscript. I have carefully read both the rebuttal and the updated version of the study.

The authors have conducted an interesting study on whether Mixture of Experts (MoE) can further improve time series foundation models, such as Moirai [1].

However, I still find the contribution of this work to be limited, given that MoE approaches have been well studied in the literature, including the field of time series analysis [2][3][4][5]. Additionally, the combination of two well established components, i.e. Moirai [1] and MoE [2][6][7][8][9], are not well studied in this manuscript. The methodology section is still very poor, particularly when compared to other impactful MoE studies [10][11]. The experimental section still leaves concerns on whether MoE approaches have been effectively combined with the Moirai [1] model. For instance, Figures 5 and 6 indicate that crucial components, such as the number of experts, were not carefully investigated. The reliance on pre-trained model-based clustering raises questions about whether the study has sufficiently explored alternatives. The rebuttal mentions that other pre-trained models [12] might also be applicable - or even more suitable - but these possibilities were not investigated. Visualisations such as in Figure 7 still fail to convey a clear and intuitive message on the expert allocation, and thus do not allow for any insightful conclusion.

Given these points, I cannot recommend accepting the study at this time, and thus leave my scores unchanged. However, I strongly encourage the authors to further improve the quality of the study, as the conceptual idea is interesting and has potential for significant impact with further refinement.

---

[1] Woo et al. "Unified Training of Universal Time Series Forecasting Transformers." ICML (2024).

[2] Zeevi et al. "Time series prediction using mixtures of experts." NeurIPS (1996).

[3] Yuksel et al. "Twenty years of mixture of experts." IEEE transactions on neural networks and learning systems (2012).

[4] Ni et al. "Mixture-of-Linear-Experts for Long-term Time Series Forecasting." AISTATS (2024).

[5] Shi et al. "Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts." arXiv (2024).

[6] Miller et al. "A mixture of experts classifier with learning based on both labelled and unlabelled data." NeurIPS (1996).

[7] Jacobs et al. "Adaptive mixtures of local experts." Neural computation (1991).

[8] Xu et al. "An alternative model for mixtures of experts." NeurIPS (1994).

[9] Hu et al. "A patient-adaptable ECG beat classifier using a mixture of experts approach." IEEE transactions on biomedical engineering (1997).

[10] Lepikhin et al. "GShard: Scaling giant models with conditional computation and automatic sharding." ICLR (2021).

[11] Riquelme et al. "Scaling vision with sparse mixture of experts." NeurIPS (2021).

[12] Ansari et al. "Chronos: Learning the language of time series." arXiv (2024).

Dear reviewer, thank you much for your reply. Please find our responses below.

[C16] However, I still find the contribution of this work to be limited, given that MoE approaches have been well studied in the literature, including the field of time series analysis [2][3][4][5].

Thanks for the comment. We respectfully disagree with this point. To clarify, we would like to restate our motivation for employing MoE. We target the challenges raising from unified time series training for foundation models. As the large and diverse time series pre-training data is highly heterogeneous, existing approaches introduce some level of model specialization to account for such heterogeneity. We identify their drawbacks and propose to automatically achieve model specialization at the token level, and we utilize the MoE concept to do this. This is a novel aspect for tackling the challenge of unified time series training for foundation models.

Regarding the references provided by the reviewer:

[2] focuses on theoretical analyses, demonstrating that a mixture of linear autoregressive models serves as a universal approximator, akin to neural networks.

[3] is a survey paper.

[4] introduces multiple linear-centric models alongside a router that weighs and combines their outputs.

[5] is a concurrent work that uses standard MoE techniques to scale up time series foundation models. The same motivation as MoE usage in large language models.

As we can see, although titles contain "mixture of experts", none of these works share the same motivation for utilizing the MoE concept as we do.

[C17] Additionally, the combination of two well established components, i.e. Moirai [1] and MoE [2][6][7][8][9], are not well studied in this manuscript.

Thank you for the comment. We respectfully disagree with the reviewer's request to evaluate the MoE methods listed in [2][6][7][8][9] with our model.

Our work targets the challenges raising from unified time series training. We utilize the MoE concept to achieve token-level model specialization and thus to account for the highly heterogeneous nature of time series data. This is actually not related to the specific designs of MoE.

Additionally, we have explored three MoE variants in our work: linear projection, linear projection with load balancing loss, and token clustering. Experimental results demonstrate that all these variants outperform prior foundation models.

Lastly, thanks for listing these MoE works. We will include and discuss the papers you suggested in the related work section of our next updated version. However, due to the last-minute of your reply, we are unable to update the current PDF.

[C18] The methodology section is still very poor, particularly when compared to other impactful MoE studies [10][11].

The initial review lists six specific points where our writing of the method section could be improved. We have carefully addressed these points in our revised submission. However, the reviewer continues to find the method section "very poor." Could you please specify which aspects remain unsatisfactory? This would help us better understand your concerns and address them effectively in the updated version of our paper.

Regarding the references ([10] and [11]) provided by the reviewer:

[10] dedicates significant content to detailing the implementation and parallel training of MoE on hardware such as TPUs. While this is a valuable discussion, it is not the primary focus of our paper.

[11] is a pioneering study on using MoE in the vision domain. In its method section, the authors first provide an overview of MoE, then explain how it can be applied in the vision domain, and finally describe the design of the gating function. This structure aligns closely with our approach in Section 3.2 of our paper.

Could the reviewer clarify the specific differences between the writing style or organization in reference [11] and our paper? This would enable us to address your concerns more thoroughly.

[C19] The experimental section still leaves concerns on whether MoE approaches have been effectively combined with the Moirai [1] model. For instance, Figures 5 and 6 indicate that crucial components, such as the number of experts, were not carefully investigated.

First, we respectfully disagree with the point "experimental section leaves concerns that whether MoE have been effectively combine with Moirai". In fact, the experimental results demonstrate the significant effectiveness of Moirai-MoE. In zero-shot forecasting, Moirai-MoE-Small achieves a 3%–14% improvement in CRPS and an 8%–16% improvement in MASE compared to all sizes of Moirai. These improvements are particularly remarkable given that Moirai-MoE-Small has only 11 million activated parameters, which is 28 times fewer than Moirai-Large.

Second, Figures 5 and 6 reveal that some experts in the deeper layers of Moirai-MoE are rarely selected. Rather than being a flaw, we think this is an intriguing finding that offers insights into the inner workings of time series foundation models. Our findings suggest that denoising processes occur progressively throughout the model. This observation is consistent with conclusions from studies [1] and [2], which indicate that, as layer depth increases, certain token vectors become redundant and are projected into the low-dimensional space defined by the top eigenvectors of the input patterns.

[1] Exploring Representations and Interventions in Time Series Foundation Models.

[2] One Fits All:Power General Time Series Analysis by Pretrained LM.

[C20] The reliance on pre-trained model-based clustering raises questions about whether the study has sufficiently explored alternatives. The rebuttal mentions that other pre-trained models [12] might also be applicable - or even more suitable - but these possibilities were not investigated.

Thanks for the comment. This concern relates to [C6]. To address it, we conducted additional experiments using the pre-trained Chronos-Small checkpoint, which matches the number of Transformer layers in Moirai-MoE-Small. The results on the Monash benchmark (the lower the better), presented below, show that the performance of this variant is comparable to Moirai-MoE, demonstrating that our proposed gating function is capable of leveraging knowledge from other foundation models.

We would like to clarify that we have never claimed the Chronos checkpoint to be more suitable.

We note that the reviewer posted this comment 4.5 hours before the deadline, leaving us insufficient time to add more experiments. Additionally, the suggestion to "sufficiently explore alternatives" lacks clarity. Could the reviewer provide more specific guidance or examples of the alternatives you believe should be explored? This would help us address your concern more effectively.

[C21] Visualisations such as in Figure 7 still fail to convey a clear and intuitive message on the expert allocation, and thus do not allow for any insightful conclusion.

Thanks for the comment. The complaint about Figure 7 aligns with the comment raised in [C13]. To address [C13], we have added experiments to investigates the relationship between raw time series observations and their corresponding expert allocations. In Appendix B.5, Figure 11, the upper subfigure presents a Traffic Hourly time series sequence with a length of 512. For enhanced visualization, the sequence is segmented using vertical dashed lines, each spanning 16 steps, which is equal to the length of a single time series token. The lower subfigure illustrates the expert allocations at shallow layers for 32 tokens derived from the 512 observations. The yellow straight line represents the specific experts selected by the token at each position. The alignment of subfigures facilitates an intuitive comparison between the time series trends and the associated expert selections.

The figure includes red square boxes to highlight time series segments exhibiting a downward trend followed by a slight upward pattern. These segments consistently correspond to the activation of two specific experts, as shown in the lower subfigure. This observation suggests that Moirai-MoE effectively captures time-based structures and demonstrates model specialization at the token level.

Dear reviewer, we would like to sincerely thank you for the time and effort put into reviewing our submission. We are grateful for your acknowledgement of the effectiveness of our approach. Please find our responses below.

[C1] MOE is a well-known technique in LLMs. Its direct application results in limited novelty for the article. Furthermore, the gating functions used in the MOE depend on the temporal embeddings from pre-trained models, which constrains the effectiveness of this approach.

Thank you for the comment. We respectfully disagree with the reviewer’s perspective. In this study, the use of MoE is specifically designed to address the heterogeneity of time series during time series foundation model pretraining. This represents a novel approach to tackling this challenge, making Moirai-MoE the first MoE-based foundation model for time series.

Regarding the proposed gating function, we argue that leveraging knowledge from a readily available pretrained model is not a weakness. On the contrary, it is a promising direction for MoE-based LLMs pretraining. For instance, notable models like Qwen1.5-MoE [1] initialize most of their parameters from its corresponding dense model Qwen1.5-7B before training the MoE model. Similarly, Nemotron-4-MoE [2] from NVIDIA presents a comparable approach.

Furthermore, we have explored various gating mechanisms in this study. The table below provides a performance comparison on the Monash benchmark. As shown, the linear-projection-based MoE already achieves competitive performance, while the token-cluster-based MoE further enhances it. This demonstrates that the effectiveness of Moirai-MoE arises from the use of MoE itself, and it is not constrained by the utilization of pretrain models' knowledge.

[1] Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters.

[2] Upcycling Large Language Models into Mixture of Experts.

[C5] In Appendix, the ablation study on patch size reveals that it significantly affects performance. This raises the question of whether multi-patch projection is indeed an effective strategy. Could the authors explain why patch size still significantly affects the model, and whether multi-patch projection remains necessary under this phenomenon?

Thank you for your thoughtful feedback. It seems there may be some misunderstanding. Moirai uses multi-patch projection for each frequency, whereas Moirai-MoE does not employ multi-patch projection. In Moirai-MoE, we employ single-patch projection and the patch size is a single hyperparameter that can be tuned. This is a common approach in patch-based models, such as TimesFM.

Regarding the choice of patch size in relation to performance, the patch size determines the time period encompassed within each token. If the patch size is too large, the linear projection layer may lack the capacity to capture the underlying patterns. Conversely, if the patch size is too small, the time series token may not contain sufficient semantic information, as highlighted in DLinear.

[C6] Could the authors provide insight into whether the token clusters generated by different time series foundational models affect the performance of MOIRAI-MOE?

Thank you for this insightful comment. We have conducted additional experiments, the details of which are outlined below. Specifically, we utilized the Chronos-Small checkpoint, as it matches the number of Transformer layers in Moirai-MoE-Small. Similar to Moirai-MoE, we performed inference on Chronos using our pretraining dataset, LOTSA, to extract the hidden states after the attention modules and generate cluster centroids at each layer. Subsequently, we pretrained a Moirai-MoE-Chronos variant using the token clusters derived from Chronos. The results on Monash, presented below, show that the performance of this variant is comparable to Moirai-MoE, demonstrating that our proposed gating function is capable of leveraging knowledge from other foundation models.

We are very grateful for your insightful comments. Your expert knowledge is helping us to strengthen the manuscript significantly. Please let us know if you have any other comments/questions.

Thank you very much for your response, which has addressed most of my concerns. However, there are still a few questions that remain unresolved:

• I would like to know the performance of MOIRAI-MOE in both few-shot and full-shot settings, and how it compares using standard benchmarking methods, such as the settings in PatchTST.

• Additionally, I am curious whether excessively long input lengths could lead to unfair comparisons. For instance, if the input length reaches as high as 5000 as shown in your table, specific models trained on such extensive data might also perform quite well, which could diminish the advantage of foundational time series models.

• Lastly, regarding the multi-patch strategy, I may not have expressed myself clearly. What I meant is that if the patch size has a significant impact on the performance of MOIRAI-MOE, does that imply that MOIRAI-MOE has not fully addressed the lack of generalizability associated with a single patch size? In that case, the original multi-patch strategy in MOIRAI should still hold positive significance.

Dear reviewer, thank you much for your reply. Please find our responses below.

[C7] I would like to know the performance of MOIRAI-MOE in both few-shot and full-shot settings, and how it compares using standard benchmarking methods, such as the settings in PatchTST.

Thanks for the comment. Knowing the performance of foundation models in fine-tuning downstream datasets under few-shot or full-shot settings is valuable; however, this requirement falls outside the scope of our study. The main goal of this work lies in developing time series foundation models that can generalize well in zero-shot forecasting scenarios [1,2,3].

And we have conducted extensive experiments to support our claim. We included 10 zero-shot datasets spanning diverse domains and sampling frequencies. We benchmarked these datasets against 11 baseline models, and, in response to the reviewer's request, added results for 6 additional baselines, bringing the total to 17. Most importantly, Moirai-MoE outperformed all 17 baselines, showcasing its superiority in zero-shot forecasting. We leave the investigation of fine-tuning foundation models into future work.

[1] Unified Training of Universal Time Series Forecasting Transformers.

[2] Chronos: Learning the Language of Time Series.

[3] A decoder-only foundation model for time-series forecasting.

[C8] I am curious whether excessively long input lengths could lead to unfair comparisons. For instance, if the input length reaches as high as 5000 as shown in your table, specific models trained on such extensive data might also perform quite well, which could diminish the advantage of foundational time series models.

Thank you for raising this point. If we understand correctly, you are suggesting that training a specialized model on a dataset with a long input length, such as 5000, could achieve strong performance or even surpass foundational models. To investigate this and also to address your concern about the potential unfair comparisons arising from different context lengths, we have conducted additional experiments: training the state-of-the-art specialized models PatchTST and iTransformers using the same context length searching strategy as in Moirai and Moirai-MoE.

The below table presents the results using 4 metrics. Please note that, for clarity, we use Range-1 to denote the searching range within prediction lengths * [2, 20], which is approximately around the values of [50, 500]. This is the range we used in our paper for specialized models. In addition, we use Range-2 to denote Moirai's strategy: searching within {1000, 2000, 3000, 4000, 5000}.

To be continued ...

Dear reviewer, we would like to express our sincere gratitude for the time you have invested in reviewing our submission, and thank you very much for appreciating the model designs of Moirai-MoE. Please find our responses below.

[C1] Please consider spending more text on the advantages of the proposed gating function.

Thanks for the comment. We have revised Section 3.2.1 of the paper to address this point, with the changes highlighted in red.

[C2] Section 4.3 does not sufficiently demonstrate the irreplaceability of token clusters in the gating function.

Thank you for pointing this out. We would like to clarify that we do not claim or emphasize the irreplaceability of token clusters in this study. The primary focus of our work is to address the challenges of achieving effective unified training for time series, and we propose leveraging MoE to enable automatic token-level model specialization.

A natural question when employing MoE is how to design the gating function. We have explored various gating mechanisms and found that token clustering performs best for time series prediction. However, we do not assert that it is irreplaceable. To further illustrate this point, the table below presents a performance comparison on the Monash benchmark. As shown, the linear-projection-based MoE can already offer competitive performance and the token-clusters-based MoE further improves it. Therefore, the superiority of Moirai-MoE stems from the use of MoE, and the token clusters method can help achieve further enhancements.

[C3] In Figure 6, from layer 4 onward, different tokens tend to route to the same experts, suggesting that the model may have more experts than necessary for practical use.

Thank you for this insightful comment. We agree with the reviewer that some experts at deep layers in Moirai-MoE are rarely selected during inference on the datasets from Monash. Pruning these underutilized experts to improve inference efficiency is an interesting direction and is left for future work.

However, as shown in Figure 4 (right part), we observe that downstream forecasting performance consistently improves with an increasing number of experts. This suggests that scaling up the number of experts during training is necessary, while we can certainly prune the experts at deep layers or perform other model quantization methods to accelerate model inference.

[C4] MOIRAI-MOE should demonstrate lower MSE as its parameter count scales up. However, Table 2 lacks sufficient evidence for this, as it only includes the MoE-Base model with 86M/935M parameters.

Thank you for your question. We have already included the results of both Moirai-MoE-Small and Moirai-MoE-Base in Figure 3 and Table 2. These results clearly demonstrate that the performance of Moirai-MoE improves as the parameter count increases.

[C5] The team has not open-sourced the code for evaluation.

Thank you for bringing up this concern. Code is available at the link: https://anonymous.4open.science/r/moirai_moe-NB88

Thank you again for your constructive feedback. We hope our clarifications address your concerns. Please let us know if you have any other comments/questions.

Dear reviewer, we would like to express our sincere thanks for the time you have taken to review our submission, and thank you very much for appreciating the effectiveness of our method and the comprehensiveness of our experimental evaluations. Please find our responses below.

[C1] The paper does not include some probabilistic metrics, such as PICP and QICE, which could provide a more comprehensive understanding of the model's probabilistic accuracy.

Thank you for this insightful comment. We have conducted evaluations using the PICP and QICE metrics, and the results are presented below. The absence of certain baselines reflects their inability to compute PICP and QICE, as they only support point forecasting. We used the geometric mean to aggregate the PICP and QICE results across datasets. The results demonstrate that Moirai-MoE achieves the best overall performance across CRPS, PICP, and QICE, highlighting its superiority in modeling both distribution accuracy and coverage. The poor performance of Chronos is likely attributed to its use of a small number of samples for probabilistic sampling.

[C2] The author mentions using the output of Moirai as cluster centroids applied in Moirai-MoE. Since Moirai itself is a foundational time series model based on a specific frequency, wouldn’t this approach introduce Moirai's frequency preferences into Moirai-MoE?

Thank you for highlighting this concern, and we apologize for not making it clearer in the paper. In our implementation, we first pretrain a Moirai model using single-patch input/output projection layers to mitigate the influence of frequencies. The cluster centroids are then derived from this pretrained model. We have revised Section 3.2.1 of the paper to address this point, with the changes highlighted in red.

Thank you again for your constructive feedback. We hope our clarifications address your concerns. Please let us know if you have any other comments/questions.