InstaTrain: Adaptive Training via Ultra-Fast Natural Annealing within Dynamical Systems

Thank you for your valuable review. We will address your questions one by one below.

Explanation of how InstaTrain addresses continuous data.

We apologize for any confusion. To clarify, the Hamiltonian used in our work is a novel modification proposed by [1], which extends the binary Ising model to support real-valued nodes. Specifically, the binary limitation of the Ising model refers to the failure of naively extending its binary nodes to real values. The Hamiltonian of binary Ising model is $H(\sigma) = -\sum_{i\neq j}^{N}{J_{ij} \sigma_i \sigma_j} - \sum_{i}^{N}{h_i \sigma_i}$. If $\sigma$ are real-valued, they evolve to $\pm\infty$ to pursue the lowest energy state, which is $-\infty$. Even if boundaries are applied to $\sigma$, $\sigma$ are only intercepted along their way to infinity, resulting in polarized nodes and essentially a binary model.

To resolve this, [1] replaces the linear term in the original Hamiltonian with a pure quadratic term, resulting in: $H_{rv} = -\sum_{i\neq j}^{N}{J_{ij} x_i x_j} + \sum_{i}^{N}{h_i x_i^2},~x_i\in R.$ $X=\left ( x_1, x_2, ......, x_N \right )$ denotes the nodes in the dynamical system, $J_{ij}$ represents the relationship between node $i$ and node $j$, and $h_i$ refers to the self-reaction strength and is forced positive. The quadratic term acts as an energy regulator, which prevents the energy from going down to $-\infty$, allowing nodes to be localized in a certain range. We have expanded the Background section (Ising-Based Hamiltonian) in the revised manuscript to enhance understanding.

The reason of excluding several widely used time series datasets.

Thanks for your valuable comments. We are currently running and comparing our method on widely used datasets and will provide the results soon. To explain why we previously did not include them, we would like to clarify that our focus is on proposing a novel online learning solution for highly dynamic environments through natural annealing within dynamical systems. Therefore, we evaluate on high-frequency datasets with rapidly evolving distributions (0.01 seconds per sample). We also want to highlight some improvements we currently finished to ensure a more comprehensive evaluation. Specifically, we add two more challenging high-frequency datasets (Ammonia and Toluene), both collected at a sampling rate of 100 Hz. We also add three new Transformer-based baselines (Autoformer [2], DLinear [3], iTransformer [4]). Our evaluation now includes a total of eleven models across five datasets and three scenarios. The Evaluation section has been updated accordingly. We attach the accuracy comparison table below for your reference.

Table: Accuracy comparison across datasets. LF / HF: Low / High Frequency. Online learning methods only have HF. In HF update, the results from the baselines are “Not Achievable” due to slow training.

Note: Results marked in italics represent “Not Achievable” due to slow training.

[1] Wu, C., Song, R., Liu, C., Yang, Y., Li, A., Huang, M., & Geng, T.. Extending Power of Nature from Binary to Real-Valued Graph Learning in Real World.

[2] Wu, H., Xu, J., Wang, J., & Long, M.. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.

[3] Zeng, A., Chen, M., Zhang, L., & Xu, Q.. Are transformers effective for time series forecasting?

[4] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M.. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting.

Dear Reviewer 3cwj,

Thank you for your thoughtful feedback and for taking the time to review our response. We sincerely appreciate your acceptance of our work. Your constructive insights have been invaluable in improving the quality of the manuscript. Please do not hesitate to let us know if there are any remaining concerns or additional details that we can address to further improve the manuscript.

Thank you once again for your thorough and valuable review.

Best regards,

The Authors

Thanks for your valuable reviews. We will address your concerns one by one in the following.

How InstaTrain would perform with higher-dimensional data, as well as any theoretical or practical limitations?

Thanks for your valuable question. To evaluate our model's performance on higher-dimensional data, we have conducted experiments on two additional high-frequency data from a chemical detection platform, both collected at a sampling rate of 100 Hz:

1. Ammonia contains time series recordings from 72 metal-oxide sensors across six different locations under consistent wind speed and operating temperatures, resulting in a (12,755 × 432) array.

2. Toluene includes time series recordings from 72 sensors at one location under ten varying conditions (two wind speeds and five operating temperatures), forming a (12,854 × 720) array.

In addition, to provide a more comprehensive comparison, we have also added three new Transformer-based baselines (Autoformer [1], DLinear [2], iTransformer [3]). Our evaluation now includes a total of eleven models across five datasets and three scenarios: static, low-frequency updates, and high-frequency updates. The Evaluation section has been updated accordingly. The results are presented in the Table 1, Figure 5, and Figure 6 of the revised manuscript. We have attached the accuracy comparison table below for your reference. Here, we present the performance of the best-performing GNN and Transformer-based model (Best Trans), a complete version of this table is provided in Table 5 in the Appendix of the revised manuscript.

There are no limitations for the dynamical system processor to scale to higher-dimensional data. It has been demonstrated that dynamical system processors inherently support spatial-temporal co-annealing capabilities [4]. Spatial co-annealing allows multiple dynamical system processors to collaboratively find the lowest energy of a joint energy landscape without slowing down the process. Temporal co-annealing ensures that the processor can efficiently find the lowest energy state by continuously searching different sub-regions of the entire energy landscape over time until convergence.

Table: Accuracy comparison across datasets. LF / HF: Low / High Frequency. Online learning methods only have HF. In HF update, the results from the baselines are “Not Achievable” due to slow training.

Note: Results marked in italics represent “Not Achievable” due to slow training.

[1] Wu, H., Xu, J., Wang, J., & Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34, 22419-22430.

[2] Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023). Are transformers effective for time series forecasting?. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 9, pp. 11121-11128).

[3] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. (2024). iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.

[4] Song, R., Wu, C., Liu, C., Li, A., Huang, M., & Geng, T. (2024). DS-GL: Advancing graph learning via harnessing nature's power within scalable dynamical systems. In Proceedings of the 51st IEEE/ACM International Symposium on Computer Architecture.

Thank you for the insightful reviews. We will answer your questions one by one in the following.

Compare with other time series prediction models

Thanks for your valuable suggestion. In response, we have added three widely used Transformer-based models in time series prediction: Autoformer [1], DLinear [2], iTransformer [3]. Additionally, we have also included two more challenging datasets: Ammonia and Toluene, to provide a more comprehensive comparison. Ammonia contains time series recordings from 72 metal-oxide sensors across six different locations under consistent wind speed and operating temperatures, resulting in a (12,755 × 432) array. Toluene includes time series recordings from 72 sensors at one location under ten varying conditions (two wind speeds and five operating temperatures), forming a (12,854 × 720) array.

Our evaluation now includes a total of eleven models across five datasets and three scenarios: static, low-frequency updates, and high-frequency updates. The evaluation section has been updated accordingly. The results are presented in the Table 1, Figure 5, and Figure 6 of the revised manuscript. We have attached the accuracy comparison table below for your reference. Here, we present the performance of the best-performing GNN and Transformer-based model (Best Trans), a complete version of this table is provided in Table 5 in the Appendix of the revised manuscript.

Table: Accuracy comparison across datasets. LF / HF: Low / High Frequency. Online learning methods only have HF. In HF update, the results from the baselines are “Not Achievable” due to slow training.

Note: Results marked in italics represent “Not Achievable” due to slow training.

Using Tables to present results

Thanks for your valuable suggestion. In the revised manuscript, we have incorporated tables to present our results more clearly and effectively. Specifically, Tables 1, 2, and 3 have been included in the main text to highlight the key findings, while Tables 4, 5, and 6 have been added to the Appendix to provide supplementary details. We greatly appreciate your insightful feedback, which has helped enhance the clarity of our work.

[1] Wu, H., Xu, J., Wang, J., & Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34, 22419-22430.

[2] Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023). Are transformers effective for time series forecasting?. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 9, pp. 11121-11128).

[3] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. (2024). iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.

Thank you for all the responses and the additional experiments. I think it's also necessary to demonstrate the model's performance on those commonly used time series datasets although they might not be of high frequency to better compare with other time series models. I will keep my scores.

Dear Reviewer hPWn,

We sincerely appreciate your feedback and valuable suggestion to include widely used time series datasets in our evaluation. To clarify, following Reviewer 3cwj's suggestion, we incorporated experiments and results on commonly used time series datasets in our revision. These additions can be found in Appendix A.3 and Table 6. We deeply apologize for not highlighting these additional experiments in our earlier response.

For your convenience, we have also included the results in Table 6 below.

We added experiments on the ETTm1, Electricity, Traffic, and Weather datasets using several SOTA models, including MegaCRN (a SOTA GNN) [1], iTransformer (a SOTA Transformer-based model) [2], NPGL [3], three SOTA online learning baselines (FSNet [4], the online learning version of PatchTST [5], OneNet [5]), and the proposed InstaTrain. Given that the highest frequency of these datasets is 10 minutes per sample, all baselines meet the high-frequency update schedule. Therefore, we evaluate the models in two scenarios: the static scenario and the high-frequency (HF) update scenario. The results are shown in the table below.

Table: MAE comparison across datasets.

In the static scenario, iTransformer, NPGL, and InstaTrain show competitive performance. Specifically, InstaTrain outperforms NPGL on the Traffic and Weather datasets while maintaining comparable performance on ETTm1 and Electricity datasets. In the high-frequency update scenario, all models show improved performance compared to the static scenario. InstaTrain achieves the lowest MAE across all datasets, outperforming baselines. Notably, FSNet exhibits a higher MAE on the Electricity dataset in the HF scenario, probably due to its lack of robustness with this dataset. The consistent superior performance of InstaTrain in the HF scenario underscores its effectiveness in leveraging model updates to enhance performance.

Thank you once again for your thoughtful and constructive comments and reviews.

Best regards,

The Authors

[1] Renhe Jiang, Zhaonan Wang, Jiawei Yong, Puneet Jeph, Quanjun Chen, Yasumasa Kobayashi, Xuan Song, Shintaro Fukushima, and Toyotaro Suzumura. Spatio-temporal meta-graph learning for traffic forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence.

[2] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.

[3] Wu, C., Song, R., Liu, C., Yang, Y., Li, A., Huang, M., & Geng, T. Extending Power of Nature from Binary to Real-Valued Graph Learning in Real World. In The Twelfth International Conference on Learning Representations.

[4] Pham, Q., Liu, C., Sahoo, D., & Hoi, S. Learning Fast and Slow for Online Time Series Forecasting. In The Eleventh International Conference on Learning Representations.

[5] Wen, Q., Chen, W., Sun, L., Zhang, Z., Wang, L., Jin, R., & Tan, T. Onenet: Enhancing time series forecasting models under concept drift by online ensembling. Advances in Neural Information Processing Systems.

We sincerely appreciate your positive feedback and your constructive suggestions. In the following, we will address your questions one by one.

Add elaboration on dataset statistics and results on more challenging datasets.

Thanks for your valuable suggestion. In response, we have expanded the Appendix of the revised manuscript to include detailed statistics of the original datasets as well as two additional, more challenging datasets. Given our focus on high-frequency data, we have incorporated two real-world datasets from a chemical detection platform, both collected at a sampling rate of 100 Hz:

1. Ammonia contains time series recordings from 72 metal-oxide sensors across six different locations under consistent wind speed and operating temperatures, resulting in a (12,755 × 432) array.

2. Toluene includes time series recordings from 72 sensors at one location under ten varying conditions (two wind speeds and five operating temperatures), forming a (12,854 × 720) array.

To provide a more comprehensive comparison, we have also added three new Transformer-based baselines (Autoformer [1], DLinear [2], iTransformer [3]). Our evaluation now includes a total of eleven models across five datasets and three scenarios: static, low-frequency updates, and high-frequency updates. The evaluation section has been updated accordingly. The results are presented in the Table 1, Figure 5, and Figure 6 of the revised manuscript. We have attached the dataset statistics table and the accuracy comparison table below for your reference. Here, we present the performance of the best-performing GNN and Transformer-based model (Best Trans), a complete version of this table is provided in Table 5 in the Appendix of the revised manuscript.

Table: Dataset Statistics

Table: Accuracy comparison across datasets. LF / HF: Low / High Frequency. Online learning methods only have HF. In HF update, the results from the baselines are “Not Achievable” due to slow training.

Note: Results marked in italics represent “Not Achievable” due to slow training.

[1] Wu, H., Xu, J., Wang, J., & Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34, 22419-22430.

[2] Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023). Are transformers effective for time series forecasting?. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 9, pp. 11121-11128).

[3] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. (2024). iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.