Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series

Thank you reviewer for the constructive feedback. Please find our response below

Q1: Adaptive patching is a well-conceived design, although I am unclear why it is termed "adaptive" when the design appears to be fixed and pre-set. Perhaps "multiscale patching" would be a more accurate descriptor.

Yes, we agree with the suggestion to rename it to multiscale patching. We initially used the term “adaptive patching” to indicate that the patch length changes across layers. However, since this change is not runtime adaptation, renaming it to multi-scale patching is indeed more accurate.

Q2: Additionally, the writing should more clearly distinguish between the designs used for pretraining and those used solely for finetuning.

Sure. All the multi-variate and exogenous related components are part of the fine-tuning flow, while the rest of the components are used in both pretraining and finetuning. We will clearly distinguish between the designs used for pretraining and those used solely for finetuning in the revised manuscript.

Q3: For the full-shot head probing, it is important to include comparisons with state-of-the-art end-to-end methods, as readers are likely interested in such comparisons.

Due to space constraints, we couldn’t add the full end-to-end methods in the main result section. However, these results are available in the appendix (Table 14) where TTM with Head probing (HP) consistently outperform other HP benchmarks and are also superior or very competitive as compared to the full end-to-end training of popular TS architectures. This demonstrates that TTM with simple head probing is both highly effective and extremely lightweight, as it avoids the need to update the backbone weights. We will move this appendix table to the main result section in the revised manuscript.

Thank you reviewer for the constructive feedback. Please find our response below

Q1: The base model TSMixer used for pre-training is not newly proposed.

Yes, you are correct that TTM builds on the TSMixer foundation. However, TSMixer doesn’t discuss the techniques required to construct a pre-trained model with effective transfer learning capabilities. To achieve this, we have introduced several innovative components on top of TSMixer, such as adaptive patching, diverse resolution sampling, and resolution prefix tuning. These enhancements are crucial for effectively handling of pre-training across datasets with varying resolutions, while maintaining minimal model size. Section 4.7 details the positive impact of these techniques through comprehensive ablation studies. With the help of these novel components and the learned pre-trained weights, we outperform TSMixer by 15% in few-shot setting, as highlighted in Table 4.

Q2: TTM necessitates to train different models for different context length and forecast length, posing some new weaknesses compared to Transformer-based pre-training models. While this paper presents forecast length adaptation strategy to handle the fixed forecast length problem, performance loss still emerges when there is a disparity between the model's forecast length and the actual forecast length.

Recent advancements in language and vision have seen a shift towards adopting small, focused foundation models over large, all-purpose ones (SLMs Vs LLMs) considering the ease and effectiveness for production deployments and high task-specific accuracy of small focused models. We have applied a similar strategy to time-series foundation models by designing TTMs that are small, focused, and tailored for specific forecasting contexts. These models are particularly well-suited for production enterprise applications, which often require minimal GPU resources for deployment and enable rapid finetuning without the risk of overfitting, challenges that are more pronounced with massive transformer models.

In addition, pretraining TTM is computationally inexpensive and can be quickly trained in less than a day, a notably faster time as compared to existing counterparts which often take several days to weeks. Hence pre-training multiple TTM has no practical challenges or limitations and can easily be achieved. We would like to highlight that, as part of this paper, we also plan to release and open-source a few pre-trained TTMs with different forecasting contexts that can widely cover most of the common enterprise use-cases.

In addition, we also support several forecast length adaptation techniques to adapt a pre-trained model for different forecast lengths with extremely minimal accuracy impact. Particularly, the pruning technique has found to be very effective (Ex. only 0.8% drop in MSE when more than 50% pruning is applied for the scenario of pruning forecast length from 720 to 336.) Please see Figure 4 for the details.

Q3: There are not sufficient experiments to prove the scaling law on pre-training models. The paper's results show that the TTM performs best with 5M parameters. Will the TTM perform better with a larger number of parameters? Showing the tradeoff between performance and cost along with parameters size increasing will facilitate understanding of the model and help choose the most suitable model size.

The scaling laws of TTMs are influenced by the amount of pretraining data used. When training the TTM on Monash data alone (~250M samples), model accuracy saturates beyond 1M parameters (this represents the TTM-Quick model referred in the paper). However, by expanding the pretraining data to 1B samples by integrating Monash with additional data sources, accuracy improvements continued with increased model size, reaching saturation around 5M parameters (the TTM-Advance Model referred to in the paper). Further increasing the model size beyond 5M did not provide additional benefits for the 1B pretraining dataset. We will include these findings in a new section on scaling laws covering various model sizes. Thank you for bringing up this important aspect.

Q4: Some parts of TTM lack ablation studies to sufficiently prove their effectiveness, such as decomposing the exogenous mixer module and decoder channel-mixing.

In general, the Exogenous Mixer and Decoder Channel-Mixing components are designed to be used together. The Exogenous Mixer captures channel correlations in the forecasts, while the Decoder Channel-Mixing captures these correlations in the past context. Using both components together allows the model to learn channel correlations from both forecasts and past contexts, providing a comprehensive view. Running one without other will have information loss in the channel-correlation. Therefore, in the ablation study, we reported them as a single component.

Thank you reviewer for the constructive feedback. Please find our response below

Q1: Adding clarification note on the TSMixer architecture

Thank you for the feedback. Sure. We will clarify the TSmixer used in this paper as compared to the other one.

Q2: Given that the pre-training datasets may include time series with varying input lengths, how do TTMs manage these differences during training?

During pretraining - we do a sliding window on every pretraining dataset to get them converted into several windows based on the TTM’s context and forecast lengths so that we can train the model with all the windows. In cases, when even a single window cannot be created as the training data is extremely small, then these datasets are skipped during pretraining. Appendix Table 8 lists the datasets which were NOT skipped as part of this filtering step.

Q3: How do TTMs handle time series with input lengths not encountered before during inference?

During inference, if the user-provided input length is greater than the TTM-configured length (K), we use the last-K time-points as the input context. On the other hand, if the user-provided length is shorter than the configured TTM length, we have 2 options: if minor adjustments are needed - then we can prepend zeros to virtually extend the length. However, for major adjustments - it is preferable to quickly pre-train another TTM with the required shorter length for enhanced accuracy. Note that, pre-training shorter context length TTM is computationally inexpensive and can be quickly trained in a matter of few hours. Even, pretraining TTM on very long context lengths can be achieved in less than a day, a notably faster time as compared to existing counterparts which often take several days to weeks. As part of this paper, we will release and open-source a few pre-trained TTMs with different forecasting contexts that can widely cover most of the common enterprise use-cases across industries.

Q4: Considering that resolution prefix embeddings are learned discretely, how do TTMs accommodate resolutions that were not seen during training when encountered during inference?

If we encounter unseen resolutions, we recommend the user to either use Out-of-Vocabulary (OOV) token configured as part of the pre-trained model to accommodate these unseen resolutions or use the TTM model pre-trained without the Resolution Prefix Tuning (RPT) module.

Q5: What criteria were used to determine the patch size for lagged exogenous features in the TTMs?

Patch size for lagged exogenous features is mostly a hyperparameter and depends on the target data-set characteristics. Please note that, Exogenous Mixer block is introduced only during the finetuning phase (on the target-data) where these parameters can easily be configured based on the target data characteristics. In general - at least a patch length of 3 or more is suggested so that we have at least one or more forward and backward lags across all channels for effective inter-channel correlation modelling of exogenous signals.

Q6: The training and evaluation of TTMs primarily focus on long time series with hundreds of input steps. Does this focus impact the performance of TTMs when applied to shorter time series, such as those with fewer than 30 time steps?

In general - TTM is pre-trained with higher context length (512, 1024 and 1536) as longer context naturally gives more information for the model to learn better and enable transfer learning. However, TTM also performs well with shorter context length too. For Example. APP, SER and CC data in our benchmarks require a shorter context length of 96 and TTM pretrained with context length 96 works pretty well and outperforms other benchmarks by a good margin (Table 6).

Thank you reviewer for the constructive feedback. Please find our response below:

Q1: Could the author intuitively explain why such few parameters can work well for time-series forecasting tasks?

There are three important design choices of TTM that greatly enhance its forecasting accuracy despite its extremely small model capacity:

1. All existing pre-trained models use a very high volume of pretraining data (TimesFM used ~300B and Moirai used 27B), hence they naturally require massive model sizes. However, as shown in Figure 3, Section 4.7, we observe that “limited” pretraining data with “high resolution diversity” greatly helps in time-series model generalization, as opposed to simply increasing the pretraining data size. This is an important observation and finding for the community that resolution diversity in pretraining data is very crucial for time-series FMs. Based on these findings, we proceed with a well-reduced dataset (1B) with high resolution diversity which naturally reduces our model size compared to counterparts needing to pretrain with several hundred billion time-series. We introduce a high diversity in our data via Diverse Resolution Sampling technique (DRS) which our counterparts fail to do.

2. Secondly, we opted for TSMixer-based models instead of transformer-based models, which further reduced the model size drastically. The TSMixer architecture has successfully established in the past that interleaving simple gated attentions with mixing components across patches, channels, and features greatly enhances forecasting accuracies with very limited model capacity, as the quadratic time-complexity of self-attentions can be entirely avoided. After TSMixer, several other mixer architectures have been published, reiterating the power of these simple architectures. Thus, avoiding complex transformer architectures further reduced our model size significantly.

3. Finally, we further increased the modeling power of TSMixer without drastically increasing its size by introducing several innovative components, such as adaptive patching/multi-scale patching, diverse resolution sampling, and resolution prefix tuning. These enhancements are crucial for effectively handling large pre-training across datasets with varying resolutions, all while keeping the model capacity very minimal.

Through these three innovative design choices, we managed to keep TTM as small as possible while outperforming state-of-the-art accuracies.

Q2: Does TTM predict the trends and seasonality well or it’s just doing conservative or simple predictions ?

Yes. Thanks for raising this important concern. Kindly refer to the pdf attached in the common rebuttal section where we have shared several zero-shot forecasting samples of TTM on various datasets, showcasing its ability to model trends and complex seasonal patterns (both real-world and synthetic sin-cos as per the request).

Q3: Does the training process involve using any synthetic data ?

In the current pre-trained models, we do not use any synthetic data. However, we augment more data via Diverse Resolution Sampling (DRS) technique to create different resolutions of existing datasets, which greatly enhances the model performance (as shown in Figure 3, Section 4.7).

Q4: Presentation may be improved, currently the presentation of figures and tables is too crowded.

Thanks for the feedback. We will address this in the final manuscript.

I appreciate the authors’ additional results addressing my concern.

The showcase plots generally meet my expectation, where the model accurately predicts the overall trend, it misses most residual details. That is, it makes conservative predictions.

Nevertheless, I am impressed by the model's performance in capturing the overall trend (seasonal trend as said by the authors) given that it uses only a few parameters.

Based on the provided results, I have adjusted the scores accordingly.