SEMPO: Lightweight Foundation Models for Time Series Forecasting

Thank you very much for the constructive comments. We are grateful for the positive comments on the technical contribution, sound experiments and well-structured presentation. Please see our detailed response below.

Weakness #1 Include full-shot experiments to evaluate model performance.

Please see our detailed response to Reviewer Y38u’s Weakness #1, where we added the full-shot experiments on four representative datasets of different scales from the TSLib benchmark.

Weakness #2 Compare with frequency debiasing model Fredformer.

Thank you for highlighting the debiasing model Fredformer. Following your suggestion, we added a comparison between SEMPO and Fredformer under 10% few-shot and full-shot settings, as summarized in Table A1 below. SEMPO demonstrates strong capabilities, consistently outperforming Fredformer under both settings in both MSE and MAE. While Fredformer shifts attention toward mid- and high-frequency components, it overlooks energy-level bias, as low-energy signals can occur across all frequency bands. In contrast, the EASD module in SEMPO suppresses dominant high-energy components to better capture subtle yet informative low-energy patterns, thereby reducing information loss and improving the forecasting generalization.

Table A1. SEMPO vs. Fredformer with 10% and 100% training data. Results are averaged over horizons of {96,192,336,720}.

Weakness #3 Overly brief description of the two-stage training paradigm in the main text.

We have explained the details of two-stage training paradigm in Lines 882-902 of Appendix B.2. For clarity, we will move this to the main text in the final version.

Weakness #4 The claimed reduction in pre-training cost needs further verification.

The pre-training data reduction we argue in the paper is mainly focused on the original data size, not considering the data size resulting from data augmentation techniques. However, even we take into account of the data size increment due to the data augmentation, our data reduction argument still holds. Specifically, random masking is a widely adopted strategy in recent time series foundation models (FMs), such as TimesFM [Ref1] and Moment [Ref2], meaning that their pre-training data size is even larger than the one, TimesFM (100B) and Moment (1.13B), we reported in Table 1 if we consider the augmented samples as part of the pre-training data size.

Unlike these FMs that merely apply masking on original samples, SEMPO duplicates each sample $N_m$ times across both energy branches, yielding a training size of $83M \times N_m \times 2$. Yet, despite this expansion, the total pre-training data of SEMPO remains substantially smaller than the billion-scale data required by other FMs such as TimesFM (100B) and Moment (1.13B). More importantly, SEMPO requires only 83M time points as original pre-training data, making it well-suited for deployment in resource-constrained environments. The cost introduced by duplication is already reflected in the reported memory usage and computational overhead. Therefore, our claim of reduced pre-training cost remains valid. We will add these discussions in the final version.

Weakness #5 Lack of comparison to TTM in zero-shot and efficiency comparisons, and results of new versions of Chronos and TimesFM.

To clarify, we have provided zero-shot comparison results with the lightweight MLP-based FM TTM in Table 9 in Appendix D.6, as explained in Lines 264-267, since Table 1 focuses on zero-shot results against Transformer-based FMs.

Following your suggestion, we have additionally included an efficiency comparison with TTM on ETTh1 and ETTm2, summarized in Table A2 below. While SEMPO is less competitive in parameter size and inference time due to its more complex Transformer-based architecture, it uses only 83M pre-training data (vs. TTM's 1B) and still achieves lower MSE on both datasets. Moreover, SEMPO outperforms TTM in 31 out of 50 cases across five diverse datasets (see Table 9 in Appendix D.6), with particularly substantial advantages on large-scale datasets such as Traffic and Electricity, demonstrating stronger knowledge transfer capability and superior sample efficiency.

Table A2. Efficiency comparison with TTM on ETTh1 and ETTm2.

To address your concern regarding the updated FMs, we added zero-shot results for the latest Chronos-Bolt and TimesFM (500M), summarized in Table A3 below for an up-to-date comparison. Across five datasets, Chronos-Bolt achieves the best performance in 16 cases, SEMPO in 12, and TimesFM (500M) in 5. We can observe that SEMPO significantly outperforms TimesFM (500M). It should be noted that, even with the small version of Chronos-Bolt (i.e., Chronos-Bolt$_S$), it is trained on nearly 100B time points with 48M parameters. In contrast, SEMPO is trained on only 83M points with just 6.5M parameters, yet it still ranks in second in terms of the overall performance. This compelling data efficiency–performance balance highlights SEMPO’s promise as an effective lightweight FM for resource-constrained environments, to which the other FMs such as Chronos-Bolt and TimesFM are infeasible.

Table A3. Zero-shot comparison on five datasets with Chronos-Bolt and TimesFM (500M).

Weakness #6 The visualization of $Z_{Lec}$ in Figure 2 appears to contain an error.

We have carefully checked Figure 2, and confirm that $Z_{Lec}$ and $Z_{Hec}$ are indeed complementary, with $Z_{Lec}=Z-Z_{Hec}$.

Weakness #7 In Figure 4(b), the low-energy feature response of Moirai seems to reflect prediction inaccuracies rather than being completely neglected.

Yes, your observation is correct. As shown by the spectral discrepancy between the prediction and ground truth, Moirai$_L$ exhibits non-negligible responses in low-energy regions, in contrast to Chronos$_S$, which largely neglects low-energy components. However, these responses in Moirai$_L$ do not consistently align with the spectral peaks of the ground truth. This indicates that while Moirai$_L$ is able to partially capture low-energy signals, it still predominantly attends to high-energy components, resulting in less accurate learning of low-energy features, such as the important positional information. We will clarify this in the final version.

References:

• [Ref1] A Decoder-only Foundation Model for Time-Series Forecasting, ICML 2024

• [Ref2] MOMENT: A Family of Open Time-series Foundation Models, ICML 2024

Thank you very much for the constructive and positive comments on our technical contribution and empirical justification. Please see our response to your comments one by one below.

Weakness #1 Lack of sufficient justification for the energy bias issue.

Thank you very much for the suggestion. We’d like to clarify that the claim regarding energy bias has been widely justified in the task-specific forecasting domain in the literature [Refs1-3], and we explained the basis of this claim in Lines 145-148. This phenomenon reflects a learning bias in Transformers, where the self-attention tends to prioritize low-energy components over low-energy ones. Such limitation has been rigorously supported by both empirical and theoretical studies. For example, Fredformer [Ref1] and FilterNet [Ref2] provide comprehensive empirical evidence showing that the dominance of low-frequency components, due to their substantial spectral energy, can lead to frequency bias, as Transformer-based forecasting models tend to disproportionately attend to these signals during learning; Amplifier [Ref3] complements this by offering a theoretical justification that frequency components with low spectral energy contribute minimally to the training loss and receive slower parameter updates compared to high-energy components.

Building upon these previous findings, and given that most existing foundation models (FMs) [Refs4-6] are built upon Transformer architectures, we empirically investigate whether energy bias persists in general-purpose forecasting domain. Our experiments on ETTh1 and Electricity datasets confirm its presence in foundation models, such as Chronos$_S$ [Ref6] and Moirai$_L$ [Ref5], as illustrated in Figures 1, 4. To further address your concern regarding this claim, we have added additional quantitative results in our response to Weakness #4, showing that other FMs like TimesFM and Timer also struggle to effectively capture low-energy frequency components.

Together, prior studies and our quantitative and qualitative results provide a rigorous justification for our claim that existing FMs exhibit energy bias.

Weakness #2 and Question #1 Unclear presentation of SEMPO’s data efficiency, such as pre-training data size and epochs.

We’d like to clarify that both the model size and pre-training data size of SEMPO and all baseline FMs have been reported in Table 1 in main text and Table 6 in Appendix C.2 (in the $(\cdot,\cdot)$ values just below the names of the models). SEMPO achieves strong zero-shot forecasting performance with just 83M time points, underscoring its remarkable data efficiency compared to baselines trained on data with hundreds of millions of time points to billions (or even hundreds of billions) of time points.

Additionally, the number of pre-training epochs have been provided in Lines 1010-1012 in Appendix C.3: in SEMPO, the energy-aware pre-training is conducted for 10 epochs, followed by 20 epochs of MoP tuning. To enable a straightforward comparison, Table A1 summarizes the pre-training epochs and data sizes of SEMPO and other four strong FMs. Notably, SEMPO requires only 83M time points, with 30 epochs of training, achieving a substantial reduction in training data and demonstrating superior data efficiency compared to baseline FMs trained on billions of time points.

Table. A1. Pre-training epochs of SEMPO and four representative FMs.

To further validate SEMPO’s data efficiency, we extended the pre-training by 5 additional epochs in either the pre-training or the tuning stage, with the results presented in Table A2 below. TTM with 20 training epochs is used as baseline. SEMPO reaches optimal performance within 30 total epochs under 83M time points, indicating that SEMPO has effectively converged and fully utilized the pre-training data.

Table. A2. Zero-shot performance with increased pre-training epochs, averaged over horizons of {96,192,336,720}.

Weakness #3 Unclear relation between pre-training and inference stages.

We’d like to clarify that we’ve provided the relationships/differences for the energy-aware pre-training stage, the MoP Tuning stage, and the inference stage in Figure 2 (Details) in Appendix B.
As mentioned in Lines 867–881 of Appendix B.1, the energy-aware pre-training does not include the MoP module, which is introduced and used only in the MoP tuning.

As also discussed in Lines 886–902 of Appendix B.2, after pre-training, the model enters the fine-tuning phase, where the model is trained on the target dataset using the same joint optimization strategy as in the MoP tuning. In this stage, only the parameters in the mixture of prompts and prediction heads are updated, while the Transformer backbone remains frozen. During zero-shot inference, the reconstruction decoder and head are removed, and there are no parameter updates in all modules. We will clarify these relationships and differences in the final version.

Weakness #4 Weak analysis of low-energy components.

We appreciate your constructive suggestion. We’d like to clarify that our analysis of low-energy components is not limited to the qualitative case study in Figure 4. In fact, we’ve also investigated SEMPO’s capability to model low-energy signals across diverse prediction lengths, with the results shown in Figure 7 in Appendix D.1.

In terms of quantitative analysis, we conducted ablation studies comparing SEMPO with its variant A.1 Multi-band Masking, which removes the energy-based partitioning mechanism (i.e., a variant that also ignores the energy bias, as what is done in existing FMs). The results across all datasets and forecasting horizons in Table 3 demonstrate the superior performance of SEMPO over this variant, in which modeling low-energy components is the only difference between SEMPO and this variant. The same observation can be found in the full ablation study results in Table 13 in Appendix D.6.

While these results provide meaningful evidence, we agree that more quantitative evaluations would strengthen the argument. In response, we justified the importance of low-energy modeling from another perspective, i.e., whether including such a component in the pre-training could improve the performance of existing FMs. The results of SEMPO and three strong FMs under two settings: with and without low-energy modeling are given in Table A3 below. Note that as the energy distribution varies across time windows and the distinction between high- and low-energy components is inherently relative, applying a unified threshold for statistical evaluation across the entire dataset is infeasible. We therefore randomly selected five batch samples from Traffic and ETTm1 and report their averaged results in Table A3.

Table A3. Impact of modeling low-energy components (LEC) on MAE across different FMs.

Two key observations can be found in Table A3. 1) Adding the low-energy modeling consistently results in very significantly decreased MAE across all FMs, highlighting the positive role these components play in enhancing the prediction accuracy; 2) SEMPO achieves the largest performance gains from incorporating low-energy components, with a 13.3% improvement on ETTm1 and 15.8% on Traffic, indicating its superior ability to effectively mitigate energy bias and model low-energy signals. We will add this quantitative evaluation to the final version.

Question #2 How it ensures routing efficiency and functional diversity？

Unlike traditional MoE that requires sparse activation over large expert networks to ensure routing efficiency and reduce computation, SEMPO adopts lightweight, learnable prompt vectors as experts. Based on these lightweight experts, SEMPO applies soft gating via a simple linear-softmax function over input tokens, enabling efficient expert selection without the need for sparse activation. These prompts are prepended to the key-value pairs in self-attention, allowing all tokens to share the same Transformer backbone. This avoids multi-branch execution, resulting in minimal computational overhead only, and well supports parameter-efficient adaptation.

To ensure functional diversity, each prompt-based expert is tailored to a specific small dataset, independently learned, and assigned input-adaptive weights via soft gating. A reparameterization strategy further transforms each expert into structured key-value pairs through a two-layer MLP, enhancing its expressiveness and promoting diverse functional roles across the expert set.

References:

• [Ref1] Fredformer: Frequency Debiased Transformer for Time Series Forecasting, KDD 2024
• [Ref2] FilterNet: Harnessing Frequency Filters for Time Series Forecasting, NeurIPS 2024.
• [Ref3] Amplifier: Bringing Attention to Neglected Low-Energy Components in Time Series Forecasting, AAAI 2025.
• [Ref4] Timer: Generative Pre-trained Transformers Are Large Time Series Models, ICML 2024
• [Ref5] Unified training of universal time series forecasting transformers, ICML 2024
• [Ref6] Chronos: Learning the language of time series, TMLR 2024

Thank you very much for the constructive comments. We are grateful for the positive comments on our methodological innovation, paper clarity and empirical justification. Please see our response to your comments one by one below.

Weakness #1 It seems the results in the experiments are from only one run. Since the masks are generated randomly, it would be more convincing if multiple runs are conducted and the standard deviations are reported

All experimental results are obtained by averaging over three runs with different random seeds. In deep time series forecasting, performance variance across seeds is typically small, and following prior work [Refs1-3], we report only the mean values. To address your concern regarding random sampling, we additionally report the standard deviation of the 5% few-shot experiments to demonstrate the stability of our method in Table A1 below.

Table. A1. Statistical results (mean ± std) of SEMPO and TTM under the 5% few-shot setting. The horizon length H is all set to 336.

The results show that both SEMPO and TTM exhibit stable performance across different seeds. Furthermore, SEMPO demonstrates its superiority by outperforming TTM in both stability and predictive accuracy in 12 out of 14 cases.

Question #1 What do Hec and Lec mean? What would you expect, if the energy-wise spectral partitioning leads to more than 2 partitions?

Hec refers to high-energy components, and Lec refers to low-energy components. If energy-wise spectral partitioning yields more than two groups, it effectively creates multiple patches along the energy axis. This is similar to the frequency patching technique. The latter offers finer granularity to capture energy patterns but introduces significant computational overhead during masking.

Question #2 In figure 4, what is "input", and what is the point of plotting "input" in the figure?

In Figure 4, "input" refers to the spectrum of the lookback window $X_{1:L}$. In time series forecasting, it is a common practice to include the lookback window in both time- and frequency-domain visualizations to present a complete forecasting sample, see similar examples in [Ref1, Ref4]. More importantly, it provides essential contextual information that helps assess whether the model's predicted spectrum preserves the spectral structure of the input and whether the model tends to emphasize certain frequency components.

Question #3 In figure 8a in Appendix, could you explain the peak at $N_m=5$? Does it make sense to show the cases with $N_m>6$?

In principle, increasing $N_m$ provides the model with more complementary frequency perspectives via multiple random samplings, facilitating the learning of diverse spectral features. However, the spike observed at $N_m=5$ is likely caused by stochastic noise rather than a stable trend. Such fluctuations may arise as redundant or conflicting frequency views introduce noise or disrupt convergence, ultimately leading to performance degradation.

Unfortunately, it does not make sense to show cases with $N_m>6$. This is because, as shown in Figure 8(a), performance does not consistently improve beyond $N_m=4$, and it may even degrade due to increased randomness in masking and potential over-fragmentation of frequency views. Meanwhile, a larger $N_m$ imposes considerable memory and computational overhead, as frequency masking is applied separately to $N_m$ duplicated inputs in both high- and low-energy branches. Considering the trade-off between performance and efficiency, we choose not to explore $N_m>6$.

Limitation Although claimed in the checklist, I failed to find a proper discussion of limitations in the conclusion.

We briefly touched on this aspect in Lines 360-362, where we discussed future directions that reflect the current limitations of SEMPO. Specifically, one limitation is that the channel-independent mechanism may hinder the modeling of critical multivariate interactions. Another is that supporting only point forecasting reduces the SEMPO’s flexibility in capturing uncertainty through probabilistic forecasting. These limitations may constrain the SEMPO’s ability to capture richer time series characteristics.

We will add all the above empirical results and clarifications in the final version and hope we have addressed your concerns.

References:

• [Ref1] Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts, ICLR 2025

• [Ref2] Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-shot Forecasting of Multivariate Time Series, NeurIPS 2024

• [Ref3] Timer: Generative Pre-trained Transformers Are Large Time Series Models, ICML 2024

• [Ref4] Fredformer: Frequency Debiased Transformer for Time Series Forecasting, KDD 2024

We sincerely appreciate your constructive and positive comments on the clear presentation and technical contribution. Please see our response to your comments one by one below.

Weakness #1 Lack of in-distribution evaluations similar to Time-MoE

Thank you very much for the insightful suggestion. To ensure fairness and enhance the comprehensiveness of our evaluation, we have added in-distribution experiments under the standard supervised setting adopted by Time-MoE. Specifically, we compare SEMPO against LLM-based time series foundation models (FMs) (S2IP-LLM and GPT4TS) as well as strong task-specific models (iTransformer, DLinear, PatchTST and TimesNet). Results on four datasets of different scales are presented in Table A1 below:

Table A1. In-domain forecasting results on four datasets. 

The results show that SEMPO consistently outperforms six advanced time series models, demonstrating that its pre-training framework effectively equips the model with strong inductive biases for downstream adaptation, despite its lightweight design. These results will be included in the final version to address this concern.

Weakness #2 Lack empirical evidence or citations to support claim on ''modeling full energy spectrum of temporal signals from pre-training data...''

Low-energy frequency components represent subtle yet informative short-term variations, such as periodicities over short durations, which serve as good indicators for forecasting. Prior studies [Refs1-3] have revealed that attending to these low-frequency components with substantial spectral energy and mitigating this energy bias are critical for enhancing performance in task-specific forecasting domain. In particular, Fredformer [Ref1] and FilterNet [Ref2] provide compelling empirical evidence that the dominance of low-frequency components with substantial spectral energy can create a bottleneck to full-spectrum information utilization, ultimately hindering forecasting performance.

Building upon this, multi-source time series pre-training can be viewed as an aggregation of learning processes across multiple individual datasets. If energy bias occurs within each dataset, it can compound during pre-training, leading to irreversible information loss. Therefore, we extend these prior findings to general-purpose forecasting domain, arguing that modeling the full energy spectrum is also essential for learning comprehensive forecasting patterns from a large collection of pre-training datasets to underpin strong generalization across datasets.

To further address your concern regarding empirical evidence, we conducted experiments comparing SEMPO with three strong FMs (Chronos$_L$, TimesFM, and Timer) under two settings: with and without low-energy components, to highlight the indispensable role of low-energy pattern modeling, please see our response to Reviewer uD87’s Weakness #4 for the detailed results and discussion. We will include the references and empirical evidence in the final version.

Weakness #3 The use of FFT and the encoder-decoder architecture with masking strategies appears standard and lacks novelty

FFT-based spectral decomposition is indeed widely adopted in time series analysis, such as in [Refs1-4]. However, it is important to clarify that these prior works primarily address the learning bias between high- and low-frequency components, a phenomenon characterized by the prioritization of low-frequency features at the expense of high-frequency features. In contrast, our work goes a step further by empirically revealing that existing time series FMs [Refs5-6] are generally biased toward high-energy components, while neglecting low-energy components that may distribute across mid-, high-, or even low-frequency bands (see Figures 1, 4, 7). The coarse-grained frequency bias correction is insufficient in this context, as it can lead to significant information loss of low-energy frequency information. To address this, we propose the novel EASD module, which performs adaptive spectral partitioning and dual-branch spectral masking to model low-energy yet informative frequency signals that are ignored in current FMs.

Additionally, most existing FMs adopt Transformer-based architectures, such as Moirai [Ref5] and Chronos [Ref6]. Leveraging a well-established encoder-decoder framework and following standard design practices does not diminish the novelty of our work in the two proposed modules. Instead, it enables us to clearly identify the above critical limitation in current Transformer-based FMs, namely energy bias. Building upon this mature architecture, we further propose the novel MoPFormer module, which enables parameter-efficient model adaptation across different datasets and domains. Together, these two innovative modules enable SEMPO to substantially reduce both the pre-training data scale and model size, while achieving strong generalization.

Weakness #4 Several formatting issues in the current draft

Thank you for pointing this out. The broken links may be due to the separation between the main text and appendix. We will address the formatting issues, including figure links and citations, in the final version to ensure preferred readability.

Question #1 Clarify whether $D_{tune}$ and $D_{test}$ are drawn from new domains or overlap with the pre-training set $T_{train}$

Following standard few-shot and zero-shot settings, $D_{tune}$ and $D_{test}$ are drawn from entirely new domains that are not presented in $T_{train}$. We will clarify this point in our final version.

Question #2 Fair and consistent visual comparison in Figure 1 (right)

We would like to clarify that Time-LLM, GPT4TS, and PatchTST are used only in the few-shot setting, while other FMs are evaluated in the zero-shot setting (see Table 1 and Table 2). Accordingly, we distinguish SEMPO (Zero) and SEMPO (Few) for fair comparison in each setting.

To address your concern regarding consistency, we will revise the figure and split it into two subplots for zero- and few-shot settings respectively in the final version.

References:

• [Ref1] Fredformer: Frequency Debiased Transformer for Time Series Forecasting, KDD 2024

• [Ref2] CATCH: Channel-aware Multivariate Time Series Anomaly Detection via Frequency Patching, ICLR 2025

• [Ref3] Towards a General Time Series Forecasting Model with Unified Representation and Adaptive Transfer, ICML 2025

• [Ref4] FilterNet: Harnessing Frequency Filters for Time Series Forecasting, NeurIPS 2024

• [Ref5] Unified training of universal time series forecasting transformers, ICML 2024

• [Ref6] Chronos: Learning the language of time series, TMLR 2024