ROSE: Register-Assisted General Time Series Forecasting with Decomposed Frequency Learning

We would like to sincerely thank Reviewer e6XG for acknowledging our presentation quality and empirical contributions, as well as the helpful comments. We have revised our paper accordingly.

Q1: Lack a clear demonstration of the effectiveness of the pre-trained model in comparison to existing time series representation learning methods.

A1:

• Compared with representation learning methods. Existing time series representation learning methods such as SimMTM [1], TS2Vec [2] and TF-C [3], primarily focus on pre-training on a single dataset and achieve in-domain prediction through fine-tuning on the same dataset. In contrast, ROSE is pre-trained on multi-source data and achieves fast adaptation in various out of domain scenarios.

  The following table illustrates the performance of ROSE in full-shot and 10% few-shot settings, in comparison with that of time series representation learning methods in full-shot settings. It can be observed that ROSE outperforms the representation learning methods in full-shot setting, and achieves a competitive performance even in 10% few-shot setting, which substantiates the effectiveness of ROSE as a pre-trained model.$\underline{\text{We have added this section to our revised paper in Appendix A.12.3}}$.

• Compared with other pre-trained models. $\underline{\text{In Table 2 in our paper}}$, we compare different pre-trained models in zero-shot setting, and ROSE demonstrates superior performance.

[1] Dong, J., Wu, H., Zhang, H., Zhang, L., Wang, J., & Long, M. (2024). Simmtm: A simple pre-training framework for masked time-series modeling. Advances in Neural Information Processing Systems, 36.

[2] Yue, Z., Wang, Y., Duan, J., Yang, T., Huang, C., Tong, Y., & Xu, B. (2022, June). Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 8, pp. 8980-8987).

[3] Zhang, X., Zhao, Z., Tsiligkaridis, T., & Zitnik, M. (2022). Self-supervised contrastive pre-training for time series via time-frequency consistency. Advances in Neural Information Processing Systems, 35, 3988-4003.

Q2: The evaluation focus on one type of downstream task, which limits the assessment of the method's robustness and generalizability.

A2: Our paper focuses on general time series forecasting as some existing models, such as Moirai [1], Timer [2] and TimesFM [3].

To verify the robustness and generalizability of ROSE in various applications of time series forecasting, we have presented the experimental results in the following tasks in the paper:

• Various downstream fine-tuning settings: including full-shot, few-shot and zero-shot settings $\underline{\text{in Section 4.1 and 4.2}}$.
• Various downstream datasets: including realistic datasets from multiple domains such as Transport, Nature, Prices and Energy, with varying numbers of channels and sampling frequencies. Detailed statistics on the downstream datasets are $\underline{\text{in Appendix A.1.2}}$.
• Various prediction lengths: including both long-term forecasting (with multiple prediction horizons: 96, 192, 336 and 720) $\underline{\text{in Appendix A.10}}$ and short-term forecasting $\underline{\text{in Appendix A.6}}$.

Q3: How to solve the problem of excessive data differences in different domains. Data with too much difference often leads to training failure.

A3: Learning a unified representation from time series data across different domains is challenging. We address this problem and avoid training failure from the following three aspects:

• Scale differences. We solve this by normalizing each pre-training dataset individually.
• Frequency distribution differences. We propose a decomposed frequency learning to solve this. we combine multi-frequency masking with reconstruction task to understand time series from both low and high frequencies, enabling the model to learn a unified representation across multi-domain datasets with varying frequency distributions.
• Domain differences. We propose the time series register to capture and store domain-specific information to solve it.

Q4: Does the dimension of the time series affect the performance.

A4: We employ the channel-independent strategy that can naturally handle time series of varying dimensions. Our experiments have included datasets with various dimensions and achieved promising results. The dimensions of different datasets are shown $\underline{\text{in Table 5 of the Appendix A.1.2}}$.

[1] Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., & Sahoo, D. (2024). Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592.

[2]Liu, Y., Zhang, H., Li, C., Huang, X., Wang, J., & Long, M. (2024). Timer: Transformers for time series analysis at scale. arXiv preprint arXiv:2402.02368.

[3] Das, A., Kong, W., Sen, R., & Zhou, Y. (2023). A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688.

Dear Reviewer e6XG,

We would like to sincerely thank you for your time and efforts in reviewing our paper.

We have made an extensive effort to try to successfully address your concerns:

1. Adding comparison with the representation learning methods;

2. Clarifying the diverse scenarios covered in our experiments;

3. Explaining the three aspects in which we address the problem of excessive data differences in different domains;

4. Explaining our implementation of channel-independent strategy to overcome the affects of dimension of the time series;

5. Making revisions to the paper and appendix accordingly.

We hope that our response can address your concerns to your satisfaction. If so, we kindly ask for your reconsideration of the score. If you have any further concerns or questions, please do not hesitate to let us know, and we will respond timely. We kindly remind you that the reviewer-author discussion phase will end soon. After that, we may not have a chance to respond to your comments.

All the best,

Authors

Dear Reviewer e6XG,

Thank you for taking the time and effort to provide a valuable review of our work. We have responded to each of your insight comments point by point.

Since the end of the discussion period is coming soon, we hope that you have had the chance to read our rebuttal. We eagerly await your feedback and are ready to respond to any remaining concerns you may have. If our rebuttal has resolved your concerns or improved your understanding of the paper, we would also be very grateful if you could reconsider the score, which would give us a greater opportunity to present this work at the conference.

Thank you once again for your time and review.

Best regards,

Authors

We would like to sincerely thank Reviewer 8yo6 for providing detailed review and insightful comments regarding the model design and empirical study . We have revised our paper accordingly.

Q1: The justification for encoder-decoder style, which (1) to some extent locks in its look-back and forecast lengths, (2) is more well received for time-series embedding (e.g. Moment [1]).

A1:

• The justification for encoder-decoder style：

  • We would like to clarify that our model is essentially an encoder-only architecture. The naming of the decoder is merely to distinguish it functionally from the Transformer encoder within the backbone. We adopt this architecture along with a masked reconstruction pre-training task, as it is demonstrated effective for time series representation learning. This is also validated by MOMENT [1] and MOIRAI [2]. Unlike MOMENT’s random masking or MOIRAI’s last masking strategy, we propose a novel frequency-based masking approach. We does not choose a decoder-only architecture because its auto-regressive prediction nature tends to result in cumulative errors.
  • To further enhance the effectiveness of this architecture for time series prediction tasks and enable few-shot and zero-shot capabilities, aiming to build a general time series forecasting model, we add prediction heads to improve prediction accuracy.

• For concern 1: The fixed look-back window and prediction lengths only apply to pre-training, and our model still supports multiple look-back window and prediction lengths in downstream tasks. Specifically, for the look-back windows and prediction lengths within the range of the pre-training settings, ROSE supports zero-shot forecasting. Moreover, in all scenarios, the model supports few-shot learning through fine-tuning.

• For concern 2: ROSE enhances prediction performance and learning efficiency in diverse downstream datasets by leveraging multi-source pre-training with both reconstruction and prediction tasks. Unlike MOMENT, which focuses more on general representation learning for various tasks, ROSE emphasizes predictive accuracy and supports both zero-shot and few-shot capabilities, making it particularly effective for forecasting. We will make it as a future work to extend ROSE to broader tasks similar to those addressed by MOMENT.

• Experiment：To demonstrate ROSE excels not only with fixed inputs, we evaluate on downstream tasks with input length significantly shorter than 512 which is used in pre-training. The following table which is updated $\underline{\text{in Table 16 of the revised paper}}$, shows the results of ROSE after fine-tuning with a look-back window of 96. Despite shorter input lengths, ROSE still achieves the state-of-the-art performance, demonstrating effective transfer of pre-trained knowledge.

  Model	ROSE	iTransformer	PatchTST	TimesNet	Dlinear	GPT4TS	$**S**^2$IP-LLM
  Metric	MSE/MAE	MSE/MAE	MSE/MAE	MSE/MAE	MSE/MAE	MSE/MAE	MSE/MAE
  ETTm1	$\underline{\text{0.389}}$/0.389	0.407/0.410	0.387/0.400	0.400/0.406	0.403/0.407	$\underline{\text{0.389}}$/$\underline{\text{0.397}}$	0.390/0.399
  ETTm2	0.272/0.321	0.288/0.332	$\underline{\text{0.281}}$/$\underline{\text{0.326}}$	0.291/0.333	0.350/0.401	0.285/0.331	0.278/0.327
  ETTh1	0.432/0.426	0.454/0.447	0.469/0.454	0.458/0.450	0.456/0.452	0.447/0.436	$\underline{\text{0.444}}$/$\underline{\text{0.431}}$
  ETTh2	0.376/0.393	0.383/0.407	0.387/0.407	0.414/0.427	0.559/0.515	0.381/0.408	$\underline{\text{0.378}}$/$\underline{\text{0.402}}$
  Weather	0.257/0.276	$\underline{\text{0.258}}$/$\underline{\text{0.278}}$	0.259/0.281	0.259/0.287	0.265/0.317	0.264/0.284	0.266/0.284
  Electricity	0.176/0.268	$\underline{\text{0.178}}$/$\underline{\text{0.270}}$	0.205/0.290	0.192/0.296	0.354/0.414	0.205/0.290	0.195/0.285
  Traffic	$\underline{\text{0.440}}$/0.276	0.428/$\underline{\text{0.282}}$	0.481/0.304	0.620/0.336	0.625/0.383	0.488/0.317	0.467/0.305

We sincerely appreciate your insightful suggestions, which have greatly helped us better clarify the model's characteristics and enhance the quality of our paper.

[1] Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., & Dubrawski, A. (2024). Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03885.

[2] Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., & Sahoo, D. (2024). Unified training of universal time series forecasting transformers. *arXiv preprint arXiv:2402.

Dear Reviewer 8yo6,

We would like to sincerely thank you for your time and efforts in reviewing our paper.

We have made an extensive effort to try to successfully address your concerns:

1. Justifying the choice of architecture;

2. Experimental proving the effectiveness of reconstructing loss is due to better representations learning rather than more labels;

3. Evaluating the model performance in more scenarios (different input/output lengths and short-term predictions in the zero-shot setting);

4. Describing the selection strategy for prediction heads;

5. Experimental proving the effectiveness of register with shorter input lengths;

6. Describing the difference between register and seasonality lookup;

7. Re-evaluating the inference time;

8. Making revisions to the paper and appendix accordingly.

We hope that our response can address your concerns to your satisfaction. If so, we kindly ask for your reconsideration of the score. If you have any further concerns or questions, please do not hesitate to let us know, and we will respond timely. We kindly remind you that the reviewer-author discussion phase will end soon. After that, we may not have a chance to respond to your comments.

All the best,

Authors

Thanks to the authors for their work. We would like to ask a question. ROSE has potential as a fundation model, and you also experimented with zero-shot prediction in your paper. But what if the prediction length of the downstream task does not coincide with the length supported by the prediction head in the model? Or what if the prediction length of the downstream task exceeds the prediction length supported by the prediction head in the model? In other words, how much performance will it affect if we use 96 head for 192/336/720 prediction tasks?

Dear reviewer 8yo6,

Thank you for taking the time and effort in providing a valuable review of our work. As the discussion period is coming to a close, we hope that you have had the chance to review our rebuttal.

If our rebuttal has resolved your concerns or improved your understanding of the paper, we would greatly appreciate it if you could reconsider your assessment and update the score accordingly. Your feedback has been incredibly helpful, and we value the opportunity to further improve the work based on your insights.

Thank you again for your thoughtful review and for considering our responses. Please feel free to reach out if you have any additional questions or require further clarifications.

Best regards,

Authors

Dear Reviewer 8yo6,

Thank you for taking the time and effort to provide a valuable review of our work. We have responded to each of your insight comments point by point.

Since the end of the discussion period is coming soon, we hope that you have had the chance to read our rebuttal. We eagerly await your feedback and are ready to respond to any remaining concerns you may have. If our rebuttal has resolved your concerns or improved your understanding of the paper, we would also be very grateful if you could reconsider the score, which would give us a greater opportunity to present this work at the conference.

Thank you once again for your time and review.

Best regards,

Authors

We would like to sincerely thank Reviewer Z8Ez for providing a detailed review and insightful comments. We have revised our paper accordingly.

Q1： Why does decomposing frequency learning help achieve unified representations, and what further analytical experiments support this?

A1:

• Design: Decomposed frequency learning helps achieve unified representations due to the following two design aspects: 1) Multi-frequency masking randomly masks high or low-frequency components of time series multiple times, which achieves decoupling of complex temporal patterns. 2) The reconstruction task enables the model to understand data from various frequency perspectives, allowing it to learn unified representations.
• Experiment: To further demonstrate the effectiveness of decomposed frequency learning in capturing unified representations, we pre-traine the model using multi-frequency masking, patch masking and multi-patch masking. We visualize the reconstruction performance of the three methods in out-of-distribution (OOD) scenarios. We put the results of the visualization $\underline{\text{in Appendix A.13.3 of our revised paper}}$. The model pre-trained with multi-frequency masking exhibits greater robustness to complex temporal patterns, confirming that decomposed frequency learning can assist in learning unified representations.

Q2: The requirement for a different new Register for each dataset during the pre-train stage.

A2: We would like to clarify that during the pre-train stage, ROSE does not require training a separate register for each dataset. Instead, a single register is used to cluster and store the domain-specific information from multi-source datasets. When a new dataset is added for pre-training, the pre-trained register can continue to be used and updated through incremental training. When a new dataset is added in fine-tuning, the pre-trained register can fine-tuning with a learnable low-rank matrix A. Two cases above are all similar to existing pre-trained models [1][2][3], without the need to start a new pre-training process.

Q3: The description of the setting for Figure 1(b) is missing.

A3: We select three datasets (Pems08, PSRA, Electricity) from transport, nature and energy domains respectively and compare the differences in hidden representations between direct transfer and adaptive transfer. Specifically, direct transfer refers to the case where domain specific information is not considered, while adaptive transfer considers domain specific information that is learned by register tokens. We visualized the output of the encoder's hidden representations using t-SNE. The description of the setting is updated $\underline{\text{in Appendix A.11}}$.

[1] Gao, S., Koker, T., Queen, O., Hartvigsen, T., Tsiligkaridis, T., & Zitnik, M. (2024). Units: Building a unified time series model. arXiv preprint arXiv:2403.00131.

[2] Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., & Sahoo, D. (2024). Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592.

[3]Liu, Y., Zhang, H., Li, C., Huang, X., Wang, J., & Long, M. (2024). Timer: Transformers for time series analysis at scale. arXiv preprint arXiv:2402.02368.

Dear Reviewer Z8Ez,

We would like to sincerely thank you for your time and efforts in reviewing our paper.

We have made an extensive effort to try to successfully address your concerns:

1. Giving further explanations and analysis experiments on the effectiveness of Decomposed Frequency Learning;

2. Clarifying the process of pre-training ROSE with register;

3. Supplementing the experimental settings for Figure 1(b);

4. Making revisions to the paper and appendix accordingly.

We hope that our response can address your concerns to your satisfaction. If so, we kindly ask for your reconsideration of the score. If you have any further concerns or questions, please do not hesitate to let us know, and we will respond timely. We kindly remind you that the reviewer-author discussion phase will end soon. After that, we may not have a chance to respond to your comments.

All the best,

Authors

Dear Reviewer Z8Ez,

Thank you for taking the time and effort to provide a valuable review of our work. We have responded to each of your insight comments point by point.

Since the end of the discussion period is coming soon, we hope that you have had the chance to read our rebuttal. We eagerly await your feedback and are ready to respond to any remaining concerns you may have. If our rebuttal has resolved your concerns or improved your understanding of the paper, we would also be very grateful if you could reconsider the score, which would give us a greater opportunity to present this work at the conference.

Thank you once again for your time and review.

Best regards,

Authors

Dear Reviewer Z8Ez,

We sincerely appreciate the time and effort you dedicated to reviewing our paper during this busy period, as well as your recognition of its strengths.

With the author/reviewer discussion phase now concluded and no additional concerns raised, we believe our rebuttal has addressed all of your comments. However, based on the scoring standards of past ICLRs, we find that our current score of 5.75 is at the borderline level. We would be deeply grateful if you could consider raising your score and giving us the opportunity to present our work at the conference.

Thank you once again for your thoughtful review and valuable feedback.

Best regards,

Authors

We would like to sincerely thank Reviewer oZ4H for acknowledging our technical novelty and effectiveness, as well as the insightful comments. We have revised our paper accordingly.

Q1: How to handle the multi-components, multi-losses and appropriately tune hyper-parameters?

A1:

• Multi-components and multi-losses:

  • The main components of ROSE include time series register, backbone, as well as prediction heads and reconstruction head.
  • We optimize these components with different loss functions. The register loss corresponds to the register. The reconstruction loss and the prediction loss correspond to the reconstruction head and the prediction heads, respectively, and they work together for the backbone.

• Multi-losses Balancing:

  • We did not use a hyper-parameter to balance the loss functions because our experiment find that the model is not sensitive to the weights of the multi-losses. In the experiment, we introduce a hyper-parameter $\lambda$ , and define $\mathcal{L} _{pretrain} = \lambda \mathcal{L} _{reconstruction} + (1 - \lambda) \mathcal{L} _{prediction} + \mathcal{L} _{register}$. Since the register loss only constrains the parameter updates of register, its gradient does not influence backbone of the model. Therefore, the register loss does not cause an imbalance in the training of the model.

  • We vary $\lambda$'s value and report results in 10% few-shot setting in the table below which are updated $\underline{\text{in Table 22 of the revised paper}}$. As ROSE is not sensitive to changes of $\lambda$, balancing the loss of the model is not challenging. Therefore, our final loss function does not contain $\lambda$.

    $\lambda$	0.2	0.4	0.6	0.8	Standard Deviation
    	MSE / MAE	MSE / MAE	MSE / MAE	MSE / MAE	MSE / MAE
    ETTh1	0.3973 / 0.4199	0.3978 / 0.4193	0.3978 / 0.4205	0.3996 / 0.4230	0.0008 / 0.0014
    ETTh2	0.3339 / 0.3790	0.3347 / 0.3802	0.3369 / 0.3822	0.3351 / 0.3830	0.0011 / 0.0016
    ETTm1	0.3500 / 0.3733	0.3512 / 0.3747	0.3492 / 0.3717	0.3479 / 0.3719	0.0012 / 0.0012
    ETTm2	0.2538 / 0.3111	0.2534 / 0.3095	0.2512 / 0.3092	0.2505 / 0.3092	0.0014 / 0.0007

• Hyper-parameters: We conduct a sensitivity analysis to tune key hyper-parameters for ROSE, including the number of masked series, thresholds' upper bound, number of register tokens, register size, and the number of selections in the Top-K strategy. Through comprehensive analysis, we determine an optimal hyper-parameter setting, as shown $\underline{\text{in Appendix A.5}}$. As the training cost of ROSE is low, such analysis is not challenging.

Q2: The necessity and effectiveness of the low rank matrix A in time series register.

A2:

• The necessity: During the pre-training stage, the domain-specific knowledge is stored in the time series register. However, different datasets also exhibit variations even at the same domain. Therefore, to better adapt to a specific downstream dataset during fine-tuning, we introduce a low-rank matrix A to further take the distinct information of the dataset into account.
• The effectiveness: $\underline{\text{In Appendix A.5, as shown in Figure 8(b)}}$, we compare the results of the model that are designed with and without the low-rank matrix A , respectively. It is found that adding the low-rank matrix effectively improves the model performance with different numbers of register tokens.

Q3: The model parameters adjusted during the fine-tuning stage.

A3: In the fine-tuning stage, we freeze the time series register, and fine-tune the learnable low-rank matrix A, the model's encoder-decoder, and the corresponding prediction head.

Dear Reviewer oZ4H,

We would like to sincerely thank you for your time and efforts in reviewing our paper.

We have made an extensive effort to try to successfully address your concerns:

1. Utilizing experiments to illustrate how to balance multi-components, multi-losses, and tune hyper-parameters;

2. Illustrating the role of matrix A;

3. Clarifying the parameters that need to be adapted during the fine-tuning phase;

4. Making revisions to the paper and appendix accordingly.

We hope that our response can address your concerns to your satisfaction. If so, we kindly ask for your reconsideration of the score. If you have any further concerns or questions, please do not hesitate to let us know, and we will respond timely. We kindly remind you that the reviewer-author discussion phase will end soon. After that, we may not have a chance to respond to your comments.

All the best,

Authors

Dear Reviewer oZ4H,

Thank you for taking the time and effort to provide a valuable review of our work. We have responded to each of your insight comments point by point.

Since the end of the discussion period is coming soon, we hope that you have had the chance to read our rebuttal. We eagerly await your feedback and are ready to respond to any remaining concerns you may have. If our rebuttal has resolved your concerns or improved your understanding of the paper, we would also be very grateful if you could reconsider the score, which would give us a greater opportunity to present this work at the conference.

Thank you once again for your time and review.

Best regards,

Authors

Dear Reviewer oZ4H,

We sincerely appreciate the time and effort you dedicated to reviewing our paper during this busy period, as well as your recognition of its strengths.

With the author/reviewer discussion phase now concluded and no additional concerns raised, we believe our rebuttal has addressed all of your comments. However, based on the scoring standards of past ICLRs, we find that our current score of 5.75 is at the borderline level. We would be deeply grateful if you could consider raising your score and giving us the opportunity to present our work at the conference.

Thank you once again for your thoughtful review and valuable feedback.

Best regards,

Authors