Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning

Q5: More variety of datasets should be included

Thanks for your advice! We additionally perform short-term forecasting over Traffic and the M4 Yearly and Covid Death benchmark borrowed from GIFT-EVAL. The results of Chronos-bolt-base are reported in the following table. Our proposed method outperforms the fine-tuning baseline on almost all the metric.

Q1: Undoubtedly, the main application of time series foundation models (TSFMs) are in the zero-shot or few-shot settings. Hence, the same experiments and analysis should be done in few-shot fine-tuning settings with different percentages of training data employed (20%, 40%, …, 100%). Then only, the efficacy of the proposed approach will be clear.

As shown in the table above, fine-tuning significantly outperforms zero-shot forecasting, and our proposed approach delivers considerable performance gains. We believe that it is time to improve full-shot performance of TSFMs for wider applications.

As for few-shot performance (with 20% training data), the following tables show the average MSE and MAE with the horizon in {96, 192, 336, 720}. FT is short for Fine-tuning, and Prune+FT is short for our proposed Prune-then-Finetune.

As shown in the table, our method can still outperforms fine-tuning all parameters in few-shot scenarios.

Q2: The pruning method is intuitively similar to dropouts as it essentially drops some network connections. Is that correct? A full comparison with dropouts should be performed for all models and datasets.

Thanks for your question! Pruning and dropout are essentially different. Dropouts randomly corrupt hidden states during training, while the inference time involves the complete network without droping any connections. By contrast, connections are dropped permanently after pruning, and we consistently use the same subnetwork during finetuning and inference.

We compare the average MSE and MAE of Chronos-bolt-base without and with dropout in the following table.

The results demonstrate that dropout can result in negative effects. We assume that the dropout technique is not suitable for time series. In computer vision tasks, it is naturally required and relatively easy to conduct classification or reconstruction even with noisy pixels. By contrast, it is not suitable to always consider all latent features in time series, which may contain noisy fluctuations or outdated information.

Q3: Some of the TSFMs suggest fine-tuning only the model’s head (e.g., the HeadProbing in TTM)

Thanks for your advice! In the following tables, we compare performance of probing and full finetuning in terms of average MSE and MAE with the horizon in {96, 192, 336, 720}. The dropout hyperparameter has been tuned based on the validation set.

Chronos-bolt-base:

TTM:

The results demonstrate that only fine-tuning the final head does not exhibit significant performance improvement. In a few cases, head probing outperforms full finetuning, while our proposed prune-then-finetune method still achieves the best performance.

As for TTM, their official code suggests finetuning full parameters to achieve higher performance on benchmarks including GIFT-EVAL. We would like to consider head probing as an approach to training efficiency.

Q5: More variety of datasets should be included

We have experimented on more benchmarks, including the Traffic benchmark and 8 datasets from GIFT-Eval, and we would like to share our new results here.

Additional results of long-term forecasting:

Additional results of short-term forecasting are reported in the following table. Metrics are calculated by following GIFT-Eval, e.g., MSE and MAE are based on raw values without z-score normalization.

Note that TTM sets a short context length and thereby a lightweight MLP Mixer network for short time series including M4 Yearly, M4 Quarterly, M4 Monthly, car_parts and restaurant. Applying pruning to this tiny model, we observe a performance decline on the validation set, i.e., the optimal pruning ratio after hyperparameter selection is 0. Thus, we only report the performance of TTM on other benchmarks as follows:

From the above results, we can see that our proposed method generalize well on these diverse data and usually outperforms the finetuning baseline, especially for Chronos.

Q4: Insignificant improvement in the MSE and MAE scores.

The reported MSE and MAE over ETT, Electricity, Weather, and Traffic benchmarks are calculated based on normalized values. Small improvements can translate to meaningful performance gains with respect to the original data quantity. For those datasets from GIFT-Eval, we calculate MSE and MAE based on raw values, and we can see that the improvements are significant. Undoubtedly, it is meaningful to downstream decision-making by improving the full-shot performance.

Moreover, our method shows consistent improvements across a wide range of benchmarks and settings, indicating strong robustness rather than performance gains limited to a few cases. In addition, the pruning process leads to a reduced model size and computation cost, making our method more efficient without sacrificing accuracy, which can benefit real-world deployment.

Q1: Few-shot settings

Dear Reviewer,

Apart from previous experiments using 20% training data, we have performed more experiments with 40% training data as you wanted.

40% training data:

The results confirm that our approach consistently generalize well. Also, it is worth mentioning we conduct experiments on smaller time series datasets, including M4 Hourly (414 samples), Covid Deaths (266 samples), Car Parts (2674 samples), Restaurant (807 samples), and US Births (7305 samples), where our approach is still effective in these few-shot forecasting tasks. Please refer to our 'Additional Experimental Results' in the previous comment.

Dear Reviewer SzHa,

Thanks for your dedication to reviewing our paper!

In our early response, we have included more experiments that demonstrate the effectiveness of our approach. We will highly appreciate it if you let us know whether your previous concerns have been adequately addressed. If you have any further questions, please do not hesitate to let us know, so that we can respond to them timely.

Thanks for your response! We have made more clarification to address your concerns.

Q5. Why choosing the above datasets

In addition to the suggested traffic dataset, we selected additional datasets from diverse domains, spanning healthcare, financial, traffic, sales, and nature, to demonstrate the robustness of our proposed solution.

Q2. Does Chronos inherently support droupout?

Chronos-bolt-base inherently supports using dropout, with a default dropout probability of 0.1 specified in their released configuration. Our Table 2 followed the dropout probabilities recommended by Chronos-bolt-base and TTM.

It is important to note that dropout and structured pruning serve distinct purposes. Dropout randomly disables neurons during training, and the all neurons still serve for inference. In contrast, our structured pruning approach aims to identify and retain a task-relevant subnetwork, which is a kind of prior knowledge acquired by pretraining, bootstrapping targeted specialization for downstream tasks. This allows the model to focus its capacity on adapting parameters that are truly relevant to the target task.

Unsurprisingly, our additional experiments with TTM show that dropout does not consistently improve fine-tuning performance, highlighting the need for our task-aware structured pruning method.

Q4. Relative improvement

The average relative improvements are listed in the following table.

It is important to note that the MSE/MAE in Table 2 are reported on normalized scales, where small differences can correspond to non-trivial improvements in the raw scale. Judging impact solely by the decimal place can be misleading in this context.

Furthermore, we focus on a broader issue: pre-trained TSFMs often underperform strong classical baselines, limiting their practical adoption. Our pruning-based approach is not just about marginal improvements, but about enabling consistent and reliable adaptation of TSFMs across diverse tasks. The win-rate is used for this purpose.

Q5 Does the pruning method mostly work on bigger models? The rationale behind choosing those datasets for TTM's expt?

Pruning is generally more effective on larger models, where activation sparsity and parameter redundancy are more prominent. In our work, we observe that pre-trained TSFMs often exhibit such sparsity, which can be disrupted if all parameters are fine-tuned indiscriminately. Our pruning method is designed to preserve these patterns, enabling effective specialization of TSFMs.

For short time series, TTM automatically selects very small model variants* with limited capacity (e.g., only ~70K parameters for Car Parts). These compact models lack sufficient redundancy. The four additional datasets we selected have longer input sequences, prompting TTM to load larger variants where pruning becomes meaningful and better aligned with our goal of leveraging sparsity.

*Technical details: Due to the MLP Mixer architecture, there is no unifed TTM to handle varying input lengths. Thus, TTM released many versions pre-trained on time series of different lengths, and its official code selects the appropriate version according to the length of a downstream time series. For example, time series of Car Parts has only 51 time steps, and TTM loads a version named "52-16-ft-l1-r2.1" with only 70,792 parameters. It is reasonable that there are very few redundant parameters in the extremely small model, which is absolutely not suitable for pruning.

Limitations (missing from the draft)

We have discussed the limitations in Appendix D, including the limited diversity of evaluation benchmarks and fine-tuning strategies. We have conducted additional experiments during the rebuttal to help address these aspects. Another limitation is that directly pruning the pre-trained model may not fully capture the most effective substructures. A potential improvement is to adopt iterative pruning [1] that prunes models warmed up with a few samples.

[1] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. ICLR'19.

We appreciate the reviewer’s feedback. We would like to clarify our experimental design choices.

Our experimental evaluation on full-shot forecasting performance follows established practices in the TSFM literature [1-7], using the widely adopted ETT, Weather, and Electricity benchmarks. These benchmark have been studied as the standard evaluation protocol in the long-term time series forecasting community over years. Achieving consistent, substantial improvements over baselines remains difficult even for state-of-the-art methods.

In response to the need for broader evaluation, we have expanded beyond the standard benchmarks by incorporating additional datasets spanning diverse domains. To the best of our knowledge, this is among the very first systematic efforts to evaluate full-shot fine-tuning performance across such a variety of datasets in the TSFM literature [1-7].

We believe our experimental evaluation appropriately demonstrates the validity and effectiveness of our approach within the established evaluation framework. The additional dataset diversity we have included represents a meaningful extension beyond standard practice.

[1] Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series. NeurIPS'24.

[2] Timer-XL: Long-Context Transformers for Unified Time Series Forecasting. ICLR'25.

[3] Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts. ICLR'25.

[4] A decoder-only foundation model for time-series forecasting. ICML'24.

[5] MOMENT: A Family of Open Time-series Foundation Models. ICML'24.

[6] Unified Training of Universal Time Series Forecasting Transformers. ICML'24 Oral.

[7] VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters. ICML'25.

Q1.1: About the average relative output norm. Why does the output norm reduce significantly compared to the input embedding.

Our inspiration comes from Figure 5c in [1]. Due to residual connections (as formulated in our Eq. 3 and Eq. 1), the relatively small norm of head outputs does not mean significant reduction on the results, since each head output is added into the input hidden states. We define the relative norm to quantify the significance of attention heads when modifying the input hidden states. The small output norm of most attention heads implies that not all heads are necessary in forecasting a specific downstream time series. As it is well accepted that multi-head attention can model heterogenous patterns, we assume that each attention head may play distinct role in handling time series hidden states from a specific domain.

[1] Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML'23.

**Q1.2 Why does large TSFM variants have better zero-shot performance even with a smaller proportion of significant heads compared to smaller variants? **

Though Moirai-large has a smaller proportion of significant heads, the absolute number of significant ones may be greater than Moirai-base, since Moirai-large has 384 heads in total and Moirai-base has 144 heads in total. Increasing model size during pretraining enhances the capacity of time series foundation models to memorize patterns more effectively, which in turn improves their zero-shot forecasting performance.

Q2: Why fine-tuning Moirai and Timer fails to outperform PatchTST?

Fine-tuning Moirai and Timer fails to outperform PatchTST because these foundation models already lag behind in zero-shot performance, as shown on the GIFT-Eval leaderboard. Compared to other state-of-the-art time series foundation models, Moirai and Timer exhibit a larger performance gap relative to PatchTST, making it more difficult to close this gap through fine-tuning alone.

Q3: include zero-shot performance in table.

Thanks for your advice! We will include it in our future manuscript. You can also refer to Figure 11-14 in our supplementary materials for a brief comparison.

Q4: Moirai_large shows an extremely high proportion of pruned-out parameters. Does the forward pass rely solely on the residual connection in these layers?

Thanks for your valuable question! When most channels in the deeper layers are pruned, these layers have minimal impact on the hidden states, meaning that the forward pass primarily relies on the residual connections to carry information forward. As a result, the final output representation is largely determined by the shallower layers, which appear sufficient for accurate prediction. As it is an interesting phenomenon that is also observed in LLMs, we may also apply early-exit techniques [2] to TSFMs, which are promising to reduce computation costs.

[2] EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism.

Q5: How does fine-tuning with dropout affect the performance?

Thanks for your suggestions! We compare the average MSE and MAE of Chronos-bolt-base without and with dropout in the following table.

The results demonstrate that dropout can result in negative effects. We assume that the dropout technique is not suitable for time series. In computer vision tasks, it is naturally required and relatively easy to conduct classification or reconstruction even with noisy pixels. By contrast, it is not suitable to always consider all latent features in time series, which may contain noisy fluctuations or outdated information.

Q6: How one epoch is defined in the experiments?

We define one epoch as a full iteration of all the training samples.

Q7: Does batch size have a significant effect?

Intuitively, a very large batch size (e.g., approaching the full data size) can capture the comprehensive patterns of the dataset, while smaller batch sizes can increase risks of mistakenly pruning importance channels. We use the batch size to control the iteration numbers and strike a balance between pruning efficiency and effectiveness.

Thanks for your positive evaluation!

When applying dropout during training, it is standard practice to scale the remaining values to preserve the expected mean value. However, this process changes the variance, and after passing through non-linear transformations, the expected mean can differ from that of a model trained without dropout. During training, the final projection layer adapts to the training-time statistics. At inference time, dropout is disabled, resulting in a mismatch between training and inference statistics, which differs from pruning. In classification tasks utilizing softmax, the model primarily focuses on relative relationships between probabilities, making it relatively insensitive to statistical discrepancies in the logits. In contrast, regression tasks are predicting absolute quantities and more sensitive to changes in the distribution of latent features. As a result, dropout does not always yield performance improvements in regression settings [1].

[1] Effect of Dropout Layer on Classical Regression Tasks. SIU 2020.

Dropout can be beneficial when training models from scratch over multiple epochs, as dropout across repeated samples enhances statistical diversity and improves generalization to various data distributions. However, pre-trained foundation models are effective at cold starting and need only one (or a few) epochs for fine-tuning. It would be better to preserve all data information to help quickly close mismatch between pre-trained knowledge and downstream data patterns.

Q1: Sensitivity to random seeds

Thanks for your advice! We prune Chronos-bolt-base in one epoch with two different random seeds, and the Jaccard similarity between the two sets of pruned channels is reported in the following table.

We do not observe significantly different subnetworks. One reason is that, during pruning, we set a large batch size (e.g., 8192 as recommended) to ensure that each sampled batch could cover diverse data patterns. Consequently, randomness mostly lies in the order of training batches, of which the total number is small (e.g., ETTh1 has 14 batches).

We will visualize the layer-wise and channel-wise difference by figures in our future manuscript.

W1: Repeated trials

We repeat runs with 5 different random seeds, and we report MSE with standard deviations in the following table.

The standard deviations are considerably small. We posit that fine-tuning foundation models have fewer randomness than traditional models since all trials are based on the same pretrained weights without random initialization.

W2: Other tasks using MOMENT

Due to the time limit, we are preparing codes and experiments, and we may provide results in the next week. Thanks!

W2: Imputation task using MOMENT

Thanks for your interest! We have performed imputation tasks with MOMENT fine-tuned by different methods. The sequence length is 512, and each patch has a 25% probability to be masked. As shown in the following table, our approach results in similar improvements. We would like to study anomaly detection tasks as a future work. Thanks for your valuable advice!

Q1 & Q3 & W2: Generalizability to Other Datasets in the financial and healthcare domains. Would pruning still help if the model’s pre-training knowledge is not directly relevant?

Thanks for your advice! As for healthcare, we additionally perform short-term forecasting over the Covid Death benchmark borrowed from GIFT-EVAL. As for financial time series, we perform short-term forecasting over the M4 Yearly benchmark. It is worth mentioning that Chronos included merely no yearly financial time series, and thus M4 Yearly is distinct to its pretraining knowledge.

The results of Chronos-bolt-base are reported in the following table. Our proposed method outperforms the fine-tuning baseline on almost all the metrics and datasets, and the improvement on Covid Death is more significant than M4 Yearly.

As for the MLP-based TTM model, its released version suitable for M4 Yearly only has 935 parameters. Applying pruning to this tiny model, we observe a performance decline on the validation set, i.e., the optimal pruning ratio after hyperparameter selection is 0. Thus, we only report the performance of TTM on Covid Deaths as follows:

From the results, we can see that pruning still brings consistent benefits even when the model's pre-training knowledge is not directly relevant. The sparsification effect essentially acts as an implicit regularizer, helping the model adapt more effectively to new domains with limited or different characteristics from pretraining data.

Q2 & W3: Sensitivity to the chosen pruning fraction and strategy? Performance under extreme pruning, e.g., pruning 90%?

That is a valuable question! We studied only moderate pruning ratios (5%-20% per epoch) in our previous hyperparameter tuning and now extend the pruning ratio to 90% according to your advice. In the following table, we report the average MSE of Chronos-bolt-base with the horizon in {96, 192, 336, 720}.

The results are not much sensitive to small pruning ratios. Performance after extreme pruning cannot get recovered after finetuning on small datasets (ETTh1 and ETTh2), where we can observe worse performance than zero-shot forecasting. However, given abundant training samples in other datasets, finetuning the model with 90% parameters pruned (from 207M to 30M) achieves the best performance. This is an interesting phenomenon that can also be observed at Moirai-large on Electricity (see Table 2 in our paper).

As for the pruning strategy, please refer to Appendix C.4 in our supplemental materials. The optimal pruning strategy siginificantly varies by models and datasets, while both pruning strategies outperforms the fine-tuning baseline in most cases.

W4: Baselines that just not updating weights

Thanks for your insightful advice! We implement the classic head probing baseline that only fine-tunes the final projection head and freezes all other layers. As for parameter-efficient fine-tuning, we report results of LoRA in Appendix C.3 in our supplementary materials, where full fine-tuning often outperforms LoRA though insignificantly. In the following table, we compare the average MSE and MAE with the horizon in {96, 192, 336, 720}.

Chronos-bolt-base:

TTM:

The results demonstrate that full finetuning is a strong baseline, and neither head probing nor LoRA exhibit significant performance improvement. In a few cases, head probing or LoRA outperforms full finetuning, while our proposed prune-then-finetune method still achieves the best performance.

Q1 & Q3 & W2: Generalizability to Other Datasets

We have just accomplished experiments on more time series that are much short and non-stationary. We would like to share our new results in the following tables, where the reported MSE and MAE are based on raw values without z-score normalization.

From the results, we can see that our proposed method generalize well on these diverse datasets and outperforms the finetuning baseline.

W1: The pruning metric adapts from LLM-Pruner

Thanks for your insightful comment! Actually, LLM-Pruner borrows the pruning metric from previous works [2] in the computer vision field. Despite the simplicity of its design, the significance of LLM-Pruner lies in that it is the first work that demonstrates the efficiency benefits of structured pruning for LLMs. We are also the first to explore pruning over time series models, while the difference is our particular focus on improving forecasting performance. Our work validates the effectiveness of pruning with only minimal and straightforward designs, overturning the conventional assumption of lossy compression in other fields. We believe that this would inspire future works to develop more advanced pruning techniques for higher predictive performance. Additionally, we present novel empirical studies that discover the sparsity and redundancy in time series foundation models. It would offer insights into why both dense models and sparse mixture-of-experts models can handle heterogeneous time series.

[2] Group Fisher Pruning for Practical Network Compression. ICML'21.