Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts

[C1] A weakness of this study is around the evaluation of the probabilistic forecasting component of the paper - which I think is more valuable than the deterministic evaluations performed. The results are indeed impressive but a statistical analysis of the probabilistic forecasting performance and characteristics would be a valuable addition to this well motivated study.

We provide standard deviations and statistical significance in Table 1 at this link: https://drive.google.com/file/d/1bwJ7dyji_OnSNkYXpA6IOpwnvF6nOZmS/view

[C2] The authors argue that different frequencies can display similar patterns and similar frequencies can display different patterns - which I agree with. However could some wavelet transformation/layer alleviate this issue within the standard architecture?

Thank you for the insightful comment. Indeed, applying a wavelet transformation in the input space can effectively help identify and distinguish patterns in time series. However, pattern specialization of Moirai-MoE are performed at each Transformer layer, and this allows our model to adaptively capture patterns at various hierarchical abstraction levels throughout the model. This hierarchical and adaptive specialization provides greater flexibility and modeling power compared to a static transformation applied at the input. Nevertheless, your suggestion of integrating wavelet transformations into the input is intriguing, and further experimentation would be valuable to empirically evaluate and compare the performance.

[C3] No adaptive patching was considered in the method - so how are different resolutions considered if at all?

Thank you for your comment. We would like to clarify that different resolutions are considered equivalent in our context, as we use a uniform patch size. The critical factor in our approach is the pattern or shape of the time series data, and the specialization of the model primarily addresses this aspect of diversity. Additionally, we have already included experiments examining the effects of different patch sizes in the appendix. The results show that the choice of patch size 16 works the best.

[C4] The authors perform mini-batch k-means clustering on the attention outputs to continuously update the cluster centroids at each layer where the cluster number is equal to the number of experts - did you observe any discrepancies doing this such as the number of clusters being significantly different to the number of experts?

Thank you for the insightful question. This question depends on the relative values of the cluster number K and the number of experts E, and we address it in two parts:

Case 1: K smaller than E — This setting is not valid in our framework. Since each cluster corresponds to an expert during MoE training, having fewer clusters than experts would result in multiple experts sharing the same cluster centroid. This leads to repeated expert selections and undermines the intended diversity and specialization of the experts.

Case 2: K larger than E — This is a more interesting scenario. For example, when K=64 and E=32, we face the challenge of mapping the 64 cluster centroids to the 32 experts. A practical and effective strategy we adopt is as follows: for each token (data point), we compute its distance to all 64 centroids, and select the top 32 centroids with the smallest average distances across tokens. These selected centroids are then used during MoE training. Despite having more clusters than experts, this approach still allows us to select the most representative centroids for expert routing.

We validated this approach experimentally. Results show that this method achieves comparable performance to the standard setting where K equals to E, indicating that even with a larger number of clusters, a careful selection mechanism can yield effective and robust MoE behavior.

[C1] While pruning is mentioned as future work, no concrete solution is provided.

Thanks for the comment. A concrete pruning solution is to first evaluate expert usage by tracking gating activations during pretraining. Identify experts with significantly fewer activations (e.g., activated less than 1% of total gating decisions) as underutilized. Completely remove these expert modules from the model architecture, thereby reducing GPU memory usage. Lastly, update the gating network accordingly to ensure it routes inputs correctly to the remaining experts.

[C2] The proposed clustering-based routing relies heavily on pre-trained MOIRAI, which introduces a dependency on the quality of the pre-trained model and dataset (LOTSA), and thus potentially limiting generalization to domains outside the pretraining distribution.

Regarding the argument about "potentially limiting generalization to domains outside the pretraining distribution": This challenge is inherent to all time series foundation models, not solely Moirai-MoE. Therefore, it underscores the importance of assembling a comprehensive and diverse time series pretraining corpus to maximize coverage of potential application domains.

Regarding the argument on the "dependency on the quality of the pretrained model": We acknowledge this dependency. Indeed, recent advancements in MoE-based Large Language Models (such as Qwen1.5-MoE) explicitly leverage pretrained dense models to introduce beneficial inductive biases, thereby enhancing model performance. Consequently, we consider this dependency a strategic strength rather than a limitation. Our work aligns with this research direction, with the assumption of a high-quality pretrained model and focusing on effectively leveraging it to achieve further performance gains.

[C3] MOIRAI-MOE contains significantly more parameters than those of MOIRAI. This increases memory demands during training and deployment. Is it possible to use model compression technology to reduce the parameters of the model?

The increase in parameter count of Moirai-MoE does not proportionally affect computational cost during inference because, at any given time, only a subset of the experts are activated. However, model compression techniques can indeed be leveraged to reduce the parameters: (1) By converting weights from full precision (e.g., FP32) to lower precision (e.g., INT8 or FP16), quantization techniques can substantially reduce memory footprint while maintaining accuracy. (2) Techniques such as knowledge distillation can transfer learned representations from a larger Moirai-MoE model to a smaller, compressed model, thus achieving efficiency without a significant loss in performance.

Implementing these compression techniques individually or in combination could mitigate memory concerns associated with Moirai-MoE. We will explore integrating these strategies into our framework and evaluate their impact on model performance and efficiency in subsequent research. Thank you for the suggestion.

[C4] When MOIRAI-MOE and MOIRAI parameters are equivalent, how does MOIRAI-MOE perform?

Our current configuration is as follows: Moirai-Base has a total of 91M parameters, while Moirai-MoE-Small has 117M parameters. To address your concern, we introduce a variant of Moirai-MoE-Small, called Moirai-MoE-Small-V2, by reducing the FFN dimension. This variant has a total parameter count of 89M, which is closely aligned with that of Moirai-Base.

We pretrain Moirai-MoE-Small-V2, evaluate it on the 29 datasets of the Monash benchmark, and compute the aggregated performance. Note that the aggregated MAE of Moirai-Base is 0.71 (according to Figure 3 in the main paper), while Moirai-MoE-Small-V2 achieves an aggregated MAE of 0.67, demonstrating superior performance.

[C5] MOIRAI-MOE still lags behind fully fine-tuned specialized models like TiDE on some datasets (e.g., ETT1 in Table 8), indicating room for improvement in extreme domain adaptation.

We believe there might be some misunderstanding. We carefully compare the performance of TiDE and Moirai-MoE on the ETT1 dataset in Table 8. The CRPS and MASE of TiDE are 1.056 and 6.898, while the CRPS and MASE of Moirai-MoE-Small are 0.288 and 1.750. Since lower values are better, Moirai-MoE-Small is actually significantly better than TiDE on this dataset.

However, to address your concern, we do find that on the Walmart dataset, TiDE performs better than Moirai-MoE-Small: TiDE scores 0.077 for CRPS and 0.814 for MASE, while Moirai-MoE-Small scores 0.090 for CRPS and 0.927 for MASE. This is a case where Moirai-MoE requires fine-tuning for better domain adaptation. After fine-tuning Moirai-MoE-Small, the results improve to 0.071 for CRPS and 0.756 for MASE, successfully surpassing TiDE.

IMPORTANT: Figures 1, 2, 3 and Tables 1, 2 are provided here: https://drive.google.com/file/d/1bwJ7dyji_OnSNkYXpA6IOpwnvF6nOZmS/view

[C1] Testing on synthetic datasets with clear clustering structures. Assess clustering uniqueness in homogeneous datasets or evaluate whether Moirai-MoE correctly identifies structures in datasets with predefined clusters.

We generate synthetic datasets with basic time series patterns to assess the ability of Moirai-MoE in a controllable way. The resulted visualization is in Figure 1.

[C2] Include zero-shot visualizations, highlight both successes and failures, explaining where Moirai fails and why

We conducted additional visualizations comparing Moirai-MoE-Small with Chronos-Small and TimesFM, as shown in Figures 2 and 3. Figure 2 illustrates failure cases for Moirai-MoE—these time series primarily exhibit trend without seasonality. In such cases, both Moirai-MoE-Small and Chronos-Small perform poorly, while TimesFM performs exceptionally well. However, we suspect possible data leakage in TimesFM, as its forecasts align almost perfectly with the future trend, which we believe could be inherently unpredictable. Figure 3 presents success cases where Moirai-MoE outperforms the others. Its forecasts are generally closer to the ground truth, whereas Chronos-Small and TimesFM tend to show larger overestimations or underestimations. Due to word count limitations at the rebuttal stage, we present only two cases here, but we will add more cases and analyses in the appendix of the manuscript.

[C3] Reporting standard deviations and statistical significance would better demonstrate the stability and reliability of the proposed approach

Standard deviations and statistical significance are in Tables 1 and 2.

[C4] Clearly explain how results were collected (e.g., best epochs for validation/test/train, last epoch for in-distribution experiments)

Moirai-MoE is pretrained for a certain number of epochs on a large time series corpus. After pretraining, the checkpoint saved from the last epoch is used to perform inference on 29 in-distribution datasets. Based on the aggregated performance, we know the optimal number of epochs required to achieve the best in-distribution performance and then we use this checkpoint for zero-shot evaluation.

[C5] Provide explanation of how "out-of-distribution" data is defined and ensure no data leakage occurs.

We follow the settings of the well-recognized time series foundation models (TSFMs), such as Moirai, Chronos, and TimesFM, the out-of-distribution data refers to those datasets (including their training, validation, and test sets) that are not included in the pretraining corpus.

[C6] The contributions are somewhat moderate. The incorporation of MoE is not a significant contribution, as it has already been used in Time-MoE.

We would like to argue our position from the following points. First, although Time-MoE has applied MoE to TSFMs, its application has not demonstrated superior effectiveness, as evidenced by its inferior zero-shot forecasting performance. In our zero-shot evaluations, its performance significantly trails behind Moirai-MoE. It does not even surpass Moirai-Large, as demonstrated in both our zero-shot evaluations and the point forecasting results over 27 datasets reported by the Chronos Team (https://aws.amazon.com/blogs/machine-learning/fast-and-accurate-zero-shot-forecasting-with-chronos-bolt-and-autogluon/). Hence, our contribution lies not merely in applying MoE to TSFMs but, importantly, in achieving notably superior performance compared to Time-MoE. Second, we have dedicated considerable effort to exploring and understanding the internal mechanisms of MoE models. This depth of analysis has not been previously addressed in the literature, including the Time-MoE study. Thus, this should be recognized as a meaningful contribution of our paper.

[C7] Have you considered InfoNCE or DeepCluster for this purpose? Could you comment about time complexity and memory complexity.

We appreciate the suggestion regarding the integration of clustering-based losses, such as InfoNCE or DeepCluster, into MoE training. We also find your provided training and inference complexity analysis reasonable. While these methods could indeed offer a more automated approach to structuring representations, their effectiveness might not be guaranteed in practice. Imposing clustering losses during MoE training could introduce optimization challenges and potentially interfere with the primary learning objectives of the model. Our current approach adopts a two-stage process. This separation offers training stability, particularly in the early stages when representations may not be sufficiently structured for effective contrastive learning. Our method can also leverage the inductive bias from the pretrained Moirai model. Nevertheless, we consider your recommendation a valuable and promising direction for future exploration.

[C1] Code repository are corrupted.

The issue appears to be related to viewing the files directly through the web interface. We verify that downloading the repository locally resolves the issue.

[C2] The paper seems to be a relatively straightforward application of existing sparse MoE techniques to MOIRAI, with some careful engineering effort to ensure that the training is successful.

Regarding the argument "the token construction process is more of an engineering exercise to make the operation and training efficient; with some careful engineering effort to ensure that the training is successful". In fact, efficiency is a crucial aspect, particularly when developing Time Series Foundation Models (TSFMs). TSFMs typically have large parameter sizes and require training on large time series corpus, making efficiency critical. Notably, the paper you referenced (Shazeer et al., 2017) uses an entire section (Section 3) to discuss efficiency challenges in training MoE models, including considerations such as batch size, model parallelism, and network bandwidth. These discussions are engineering-focused—arguably even more so than our paper—but does that diminish their importance or exclude them from being considered technical contributions?

Regarding the argument "the paper seems to be a relatively straightforward application of existing sparse MoE techniques". While it may sound easy to apply existing MoE techniques to TSFMs, making these techniques effective in practice is challenging. Our work represents a pioneering effort in successfully adapting sparse MoE to TSFMs. How do we define success in this context? An intuitive and primary criterion for success is the downstream model performance. For instance, the concurrent work Time-MoE, can hardly be considered successful, as its performance significantly trails behind Moirai-MoE. Notably, Time-MoE does not even surpass Moirai-Large, as demonstrated in both our zero-shot evaluations and the point forecasting results over 27 datasets reported by the Chronos Team (https://aws.amazon.com/blogs/machine-learning/fast-and-accurate-zero-shot-forecasting-with-chronos-bolt-and-autogluon/). In summary, while a straightforward application of existing MoE techniques to TSFMs may seem easy, achieving SOTA performance is technically non-trivial and depends critically on model design.

Finally, as you acknowledged in the strengths part, we have put significant effort into the behavior of MoE models. This area of investigation has not been explored in the literature, including the Time-MoE work. Thus, this should be recognized as a meaningful contribution of our paper.

[C3] Cannot detect a significant difference between the proposed gating and the procedure in Shazeer et al, 2017. A more correct claim would be “we introduce a novel procedure for pre-training the gating function in a standard sparse mixture-of-experts model”.

Your suggested point is reasonable in certain respects; however, we would like to emphasize that the paper you mentioned (Shazeer et al., 2017) employs a linear projection as its gating, which relies on a randomly initialized, trainable weight matrix for expert assignment. In contrast, Moirai-MoE utilizes a gating function that is non-trainable and initialized based on clustering results from a pretrained model, not randomly. We believe this distinction marks a departure from the linear projection gating, effectively constituting a new gating function. We would appreciate further clarification regarding which aspects you consider identical between these two gating functions.

[C4] No confidence intervals and statistical tests.

Standard deviations and statistical significance are in Table 1: https://drive.google.com/file/d/1bwJ7dyji_OnSNkYXpA6IOpwnvF6nOZmS/view

[C5] Although the token clusters approach achieves the best performance, these results suggest that the training is not operating correctly.

Thank you for your comment. We do not believe our training is incorrect. Our routing mechanism is based on clustering results derived from a pretrained Moirai model's representations. If certain clusters exhibit similarity, the selection of experts would be naturally constrained, resulting in some experts being underutilized. Thus, the question here pertains to the representation similarity inherent in pretrained TSFMs. Existing research has investigated representation similarities within TSFMs. A recent study (https://arxiv.org/pdf/2409.12915v2) uses Centered Kernel Alignment to measure representation similarity, highlighting clear redundancy in TSFMs such as Chronos and Moirai. Additionally, another study (https://arxiv.org/pdf/2302.11939v2), specifically Section 7, reports that within-layer token similarity tends to increase in deeper Transformer layers. This finding aligns closely with our own observations in Moirai-MoE, where fewer experts are activated due to increased token similarities, particularly in deeper layers.