Towards a General Time Series Forecasting Model with Unified Representation and Adaptive Transfer

We would like to sincerely thank Reviewer xtuG for acknowledging our presentation quality and empirical contributions, as well as the helpful comments. We will revise our paper accordingly.

Q1: How is the zero-shot experiment of Moment conducted?

A1:

• Moment[1] mentioned in the paper that it can be used in a zero-shot setting by retaining its reconstruction head.
• Experiment details: The zero-shot experiment of Moment is designed based on the reconstruction task in the pre-training phase. Specifically, to ensure consistency between pre-training (reconstruction) and downstream prediction tasks, we actively mask the time periods to be predicted in the input sequence, and directly use the model's reconstruction output for this part as the prediction value. We will add these descriptions to the revised paper.

Q2: Why does the efficiency analysis consider training time for full-shot models while comparing them to zero-shot models?

A2: We include the training time of full-shot models in the efficiency analysis mainly for practical application considerations. When handling a specific downstream task, actual users can choose full-shot models which need to be trained from scratch for optimal performance, or zero-shot models that can predict directly without extra training. Comparing the end-to-end efficiency of both is more meaningful for users.

Q3: What is the efficiency of the fine-tuned ReadyTS in the full-shot setting? How does its computational cost compare to other full-shot models?

A3: As suggested, we have supplemented the efficiency comparison between ReadyTS and other full-shot models. As shown in the table below, ReadyTS has competitive efficiency compared with full-shot models. This is because, due to the pre-training, ReadyTS requires fewer epochs to achieve optimal performance.

Q4: What is the definition of "averaged time" in Table 8?

A4: For the foundation models, "averaged time" is the inference time averaged over running model on the entire test set, and for the specific models, we also need to add the time it takes to train on the training set (since they can't make predictions on a test set directly).

Q5: It is unclear how the "Aggregator" in Equation 4 is defined.

A5: The aggregator is the averaging operation along the dimension of $K_f$. For example, the outputs of the encoder have the shape [ $K_f$, batch_size, n_vars, d_model, num_patch], we aggregate them to the shape [1, batch_size, n_vars, d_model, num_patch].

Q6: When passing the representation to the linear head, do you flatten it or compute the mean over certain dimensions?

A6: For the prediction head, we flatten the representation as previous classic works [2] before inputting it for prediction. For example, the representation of the shape [batchsize, n_vars, d_model, num_patch] is flattened out as the shape [batchsize, n_vars, d_model*num_patch].

Q7: Sensitivity of context length on downstream task. Whether it can still achieve strong performance when the downstream task involves a different context length ?

A7: ReadyTS is not sensitive to the context length on downstream tasks. To demonstrate this, we use 96 as the downstream input length that is significantly different from 512 as used in pre-training. The following table shows the results of ReadyTS after fine-tuning with a look-back window of 96. Despite shorter input lengths, ReadyTS still achieves state-of-the-art performance, demonstrating effective transfer of pre-trained knowledge.

[1] Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., & Dubrawski, A. (2024). Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03885.

[2] Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. (2022). A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730.

We would like to sincerely thank Reviewer DsF3 for providing a detailed review and insightful comments. We will revise our paper accordingly.

Q1: Lack of some baselines

A1: Based on your suggestions, we add some recent baselines: LPTM, Time-MOE-large and Chronos-bolt-base. As shown in the table below, ReadyTS also exhibited competitive performance compared to these baselines.

Q2: Runtime complexity and pre-train compute resources can be mentioned.

A2:

• Runtime complexity : In Section 4.3, we compare the efficiency of the foundation model by computing the inference time. Additionally, the following table shows the memory usage for different foundation models (using the ETTh2 dataset as an example with a batch size of 1).

• Pre-train compute resources: In the pre-training stage, we used 2 NVIDIA A800 80GB GPUs for only about 20 hours of training.

We would like to sincerely thank Reviewer 3YLu for providing detailed review and insightful comments regarding the model design and empirical study. We will revise our paper accordingly.

Q1: Resharpening of the text

A1: Thank you for your valuable suggestions on the presentation of the article.

• We would like to clarify that the generalized representations refer to those learned by time series pre-training frameworks, not LLMs. Thus, we update the description as follows. "However, as shown in Figure 1(a), existing time series pre-training frameworks focus mainly on learning generalized time series representation during pre-training and overlook domain-specific representation. Such direct transferring the generalized time series representation to different target domains limits the effectiveness of these frameworks in specific downstream tasks."
• We would like to clarify that since we have discussed the intuition of learning generalized time series representation in Section 3.2 Decomposed frequency learning, we simply state "By decomposed frequency learning, we can obtain the general representations" at the beginning of Section 3.3. To further clarify the description, we thus update the sentence as "As described in Sec. 3.2, we can obtain the general representations by decomposed frequency learning."

Q2: Limited evaluation benchmark for a pre-trained time series model.

A2: We add a more comprehensive fev benchmark you mentioned. The fev benchmark focuses on the model's short-term prediction ability, while we design ReadyTS that shows more expertise at long-term predictions of 96-720 steps. Nevertheless, ReadyTS still exhibited competitive performance and superior efficiency compared to other foundation models.

Q3: Selection of pre-trained models in comparision.

A3:

• We select five foundation models for comparison in zero-shot setting, including Timer-67M, MOIRAI-large, TimesFM-200M, Chronos-small and Moment-base.
• We appreciate your understanding that these are concurrent work, and we will provide the full results of all models in the revised paper.
• Due to the low inference efficiency of Chronos-large, we chose the small version for comparison. Subsequently, we add Chronos-large as a baseline in the fev benchmark.

Q4: The multi-loss setting might be difficult to balance.

A4: Due to space limitations, we respectfully refer you to check A2 in Reviewer EPkr.

Q5: Textual errors in the paper.

A5: The result of Chronos in Table2 - Traffic should be 0.615, and ReadyTS achieves the best result. We will correct this in the revised paper. We will also remove the placeholder text from Appendix A.10.4 and add explanatory text to A.10.1, 10.2, and 10.4 to correspond to the tables.

Q6: Do the individual masked sequences of one time series get processed independently by the transformer encoder?

A6: Yes.

Q7: How to handle different lengths of time series when the full time series is projected into the embedding $x_e$, which is used in TS-Register?

A7:

• The register can handle input time series with different lengths without failing. To adapt to different input lengths, we create a new linear projection layer for the new input length and update its parameters during fine-tuning.
• Experiment: We validate the effectiveness of the register under an input length of 96 which is different from 512. The average results of all predicted lengths in the table below demonstrate that the effectiveness of the register is maintained.

Q8:The gradients of the prediction heads.

A8: As shown in Figure 2, the gradients of the prediction loss only skip the decoder part and still affect the rest of the model.

Q9:The reason of distinguish between reconstruction and prediction decoder.

A9: In forward-propagation, copying the parameters of the reconstruction decoder to the prediction decoder is same as using one decoder for both tasks. However, in back-propagation, the gradient of the prediction loss skip the decoder, while the reconstruction loss is propagated normally. The reason we designed two decoders is that we do two tasks in pre-training. However, considering the convenience and effectiveness of training, we share the parameters of the two decoders and let the gradients of the prediction loss skip the decoder.

We would like to sincerely thank Reviewer EPkr for acknowledging our technical novelty and effectiveness, as well as the insightful comments. We will revise our paper accordingly.

Q1: How is the register initialized? Is it trained together with the foundation model? Why is the prediction task needed during the pre-training?

A1:

• Register initialization: the vectors in register are randomly initialized from a standard normal distribution $\mathcal{N}(0,1)$.
• The register is trained together with the foundation model, while the gradient of register loss does not affect the backbone.
• To further enhance the effectiveness of this architecture for time series prediction tasks and enable few-shot and zero-shot capabilities, aiming to build a general time series forecasting model, we add prediction heads to improve prediction accuracy.

Q2: In Equation (12), are there any trade-off hyper-parameters needed between these different losses to balance their effects? It may need some experiments regarding this.

A2:

• We did not use a hyper-parameter to balance the loss functions because our experiments found that the model is not sensitive to the weights of the multi-losses. In the experiment, we introduce a hyper-parameter $\lambda$ , and define $\mathcal{L} _{\text{pretrain}} = \lambda \mathcal{L} _{\text{reconstruction}} + (1 - \lambda) \mathcal{L} _{\text{prediction}} + \mathcal{L} _{\text{register}}$. Since the register loss only constrains the parameter updates of the register, its gradient does not influence the backbone of the model. Therefore, the register loss does not cause an imbalance in the training of the model.
• We vary $\lambda$'s value and report results in the table below. As ReadyTS is not sensitive to changes of $\lambda$, balancing the loss of the model is not challenging. Therefore, our final loss function does not contain $\lambda$.

Q3: Figure 1(b) needs more explanations. How are the hidden representations extracted in direct transfer and adaptive transfer respectively?

A3: We select three datasets (Pems08, PSRA, Electricity) from transport, nature and energy domains respectively and compare the differences in hidden representations between direct transfer and adaptive transfer. Specifically, direct transfer refers to the case where domain specific information is not considered, while adaptive transfer considers domain specific information that is learned by register tokens. We visualized the output of the encoder's hidden representations using t-SNE. The description of the setting would be updated in the revised paper.