Scaling Law for Time Series Forecasting

Dear Reviewer 7rY6,

Thank you for a detailed review! We truly appreciate your reviews and questions, especially for our theories, which aim our theory towards an omnipotent theory. We would like to briefly summarize your concerns, and our rebuttal and then expand upon this summary.

1.Theory Assumptions

1.1. Your view: Current Assumptions lead to close-to-linear relationships.

Response:

1.1.1.Our current assumptions is not equivalent to linear, and we can further weaken our assumptions while deriving similar results, (which would be more complicated a process hence we did not choose to do so in our original submission, which is presented later in this rebuttal).

1.1.2. We chose stronger assumptions for simplicity of derivation, which may lead to closer to linear models. We would like to mention that although counter-intuitively, linear models are indeed important models in time series forecasting, represented by a series of important experimental and theoretical works for time series[1,5,6].

We make these assumptions, not to include every possible scenario, but to focus on the most relevant properties related to the research problem.

1.2. Your view: Current assumptions lead to autoregressive behavior.

Response:

1.2.1. Our theory utilizes Conditioned Expectation, which actually still holds even in cases you refer to as Nebensatz. We will use the exact example of Germany to show how it works.

1.2.2. The task of Time Series Forecasting itself has assumed time series follows at least partially autoregressive behaviors, and the success of data-driven methods has validated this assumption.

1.3. Your view: Zip-f is too strong an assumption.

Response:

1.3.1. In our paper we have proven it under CI case experimentally using PCA, presented in Figure 4 and Figure 8 in the original submission.

1.3.2. We further conduct PCA on intermediate vectors with iTransformer, which represents the CD case. We appreciate your advice, hence we use p-value to show fitness. Please refer to Figure 1 in Extra Page PDF.

1.4. Your view: The claim in line 85 (the intrinsic dimension of a sequence is proportional to its length) is wrong.

Our response:

1.4.1. This is an empirical result of previous work[7] for chaotic systems, which can not be denied by manually constructed counter-examples. (E.g. It does not hold for one to claim Zipf-law for vocabularies in NLP being wrong by constructing counter-examples like '\n'.join(['I like {}'.format(n) for n in ALL_NOUNS]).)

1.5. Notions and Typos:

Thank you for pointing these out! We would check them and further polish our paper.

2. Double-descent curve and ever-decreasing loss.

We have never claimed that our theory covers all aspects and all phenomena in time series forecasting tasks. What we are focusing on theoretically is to explain the impact of look-back horizon on scaling behaviors (as well as dataset size and model size), which is not yet well-studied. In Figure 2 of our original submission, we posited that the observed increase in loss could be attributed to overfitting. We further conduct experiments to validate the impact of dataset size, model size and look-back horizon on scaling behaviors for time series forecasting, which was not validated before. (Please refer to global rebuttal).

More details.

1.1.1 Weaker Assumptions

The Quasi-isometric assumption can be replaced with this Inverse Lipschitz assumption: $\\phi^{-1}$ would be $K_I$-Lipschitz under L2 norm. That is:

$$\\forall x,y \\in \\mathcal{M}(L), \\| \\phi^{-1}(x)-\\phi^{-1}(y)\\|_2 \\leq K_I\\| x-y\\|_2.$$

We further make two assumptions:

Causality. We assume existing an optimal model $F[S]$ to predict the next $S$ frames given previous $h\rightarrow\infty$ frames, so that the error only originates from the intrinsic noise.

\\exists F[S]: \\mathcal{M}\rightarrow \\mathcal{M}(S), s.t. \\lim\\limits_{h\\rightarrow \\infty} \\mathbb{P}(y\\mid x_{-h:0}) = (1-\\eta)\\delta(F[S]\(x_{-h:0})) + \\eta \\mathcal{N}(F[S]\(x_{-h:0}), \\Sigma_S)

Where $\\eta$ stands for the noise ratio in the system and we use $\\mathcal{N}(\\mu, \\Sigma)$ to represent a normal distribution with mean $\\mu$ and covariance $\\Sigma$ here.

Uniform Sampling noise: When drawing a single length-$L$ sample, we assume that the sampling noise is uniform in each direction in $\\mathcal{L}$,

$$\mathbb{E}_{x[-H:S]\in \mathcal{O}(H+S)}[\|\phi^{-1}(m(y_i)) - \phi^{-1}(y_o)\|_2] \leq K_I e_m$$

And from assumption 1 the second term is bounded by $e(S)$. We can derive that:

$$L_{\text{ori}} \leq K_I e_m + e(S)$$

which shows loss in original space is linearly bounded by loss in the intrinsic space. W.l.o.g we may only study loss in the intrinsic space.

Because this rebuttal section has character limitation, we would like to present further deduction in detail later in the discussion period.

1.2.1. How it works for future-dependency like Nebensatz.

Take two phrases as an example:

a.Ich wusste nicht, dass er Hamburger essen wollte.

b.Ich wusste nicht, dass er Bier trinken wollte.

If P(essen | Ich wusste nicht, dass er) = P(trinken |Ich wusste nicht, dass er ) = 1/2, and P(Hamburger | Ich wusste nicht, dass er __ essen wollte) = P(Bier | Ich wusste nicht, dass er __ trinken wollte) = 1.

It is possible in practice that we do not know what comes next had we not seeing 'essen' or 'trinken' no matter how many words before that we see, but this uncertainty has already been modeled into Bayesian Loss for $M(\infty)$ and would be absorbed in the term $\\eta* tr(\\Sigma_S)$ in our bound for $L_{Bayesian}$, which will be further absorbed into the term $\\eta \\sigma_M^2 S^2 d_I(S)$ in the last formula on Page4 of our original Submission, only adding a constant to the final loss.

We have listed papers cited above in the global rebuttal. They could possibly act as supplement to our reply for your detailed concerns.

We'll present a more accurate set of assumption with weaker assumptions and how to derive similar result with these assumptions. Note that in our original submission we used stronger assumptions. We do not expect our theory to cover every aspect and explain every scenario, but to focus on the most important properties related to our research problem; and the following are a more complicated version, for your reference, in which we use much weaker assumptions to replace the quasi-isometry assumption.

The revised assumptions are as follows:

1.Information-preserving: Intuitively speaking, we should be able to recover the real sequence from its corresponding intrinsic vector with the expected error bounded by a small constant value. Formally we can state this as follows:

Exists a mapping $\phi$ from the original length-$L$ sequence space $\mathcal{O}(L)$ to $\mathcal{M}(L)$, an inverse mapping $\phi^{-1}:\mathcal{M}(L)\to \mathcal{O}(L)$ and a small constant $e(L)$ related to $L$ so that for any $\mathbb{E}_{x\sim \mathcal{O}(L)}\|x-\phi^{-1}(\phi(x))\|_2^2\leq e(L)$.

2.Inverse Lipschitz: $\phi^{-1}$ should be $K_I$-Lipschitz under L2 norm. That is:

$$\forall x,y \in \mathcal{M}(L), \|\phi^{-1}(x)-\phi^{-1}(y)\|_2 \leq K_I \|x-y\|_2$$

3.Bounded: Unchanged

4.Isomorphism: Unchanged

5.Linear Truncation: Unchanged

6.Causality: We assume there exists an optimal model $F[S]$ to predict the next $S$ frames given previous $h\rightarrow\infty$ frames, so that the error only originates from the intrinsic noise.

\exists F[S]: \mathcal{M}\to \mathcal{M}(S)\, s.t. \lim\limits_{h\to \infty} \mathbb{P}(y\mid x_{-h:0}) = (1-\eta)\delta(F[S]\(x_{-h:0})) + \eta \mathcal{N}(F[S]\(x_{-h:0}), \Sigma_S),

Where $\eta$ stands for the noise ratio in the system and $\mathcal{N}(\mu, \Sigma)$ represents a normal distribution with mean $\mu$ and covariance $\Sigma$.

7.Uniform Sampling noise: When drawing a single length-$L$ sample, we assume that the sampling noise is uniform in each direction in $\mathcal{L}$,

Assumptions 1 and 2 ensure that if we predict the intrinsic vector accurately, we can predict the original time series well. Thus we may only consider the task to predict a vector in $\mathcal{M}(S)$ given the vector corresponding to its previous $H$ frames in $\mathcal{M}(H)$, which justifies the task formulation.

A formal deduction is shown as follows:

Proof.

Consider we are predicting $x[0:S]$ from $x[-H:0]$, let $y_i\in \mathcal{M}(H)$ be the intrinsic vector of $x[-H:0]$ and $y_o\in \mathcal{M}(S)$ be the intrinsic vector of $x[0:S]$ (the true intrinsic vector). If we have a model $m$ so that:

$$\mathbb{E}_{x[-H:S]\in \mathcal{O}(H+S)}[\|m(y_i)-y_o\|_2] \leq e_m$$

where $e_m$ represents the expected error in the intrinsic space. Then, from assumption 2 we have:

$$\mathbb{E}_{x[-H:S]\in \mathcal{O}(H+S)}[\|\phi^{-1}(m(y_i)) - \phi^{-1}(y_o)\|_2] \leq K_I e_m$$

and from assumption 1 we know that:

$$L_{\text{ori}} \leq K_I e_m + e(S)$$

Hence, the loss in the original space should be:

$$L_{ori} = \mathbb{E}_{x[-H:S]\in \mathcal{O}(H+S)}[\|\phi^{-1}(m(y_i)) - x[0:S]\|_2] \\\\$$ = \mathbb{E}_{x[-H:S]\in \mathcal{O}\(H+S)} \[\|\phi^{-1}(m(y_i)) - \phi^{-1}(y_o)\|_2] $$\text{ } \text{ }\text{ } + \mathbb{E}_{x[0:S]\in\mathcal{O}(S)}[\|\phi^{-1}(y_o)-x[0:S]\|_2]$$ $$\leq K_I e_m + e(S)$$

which shows that the loss in the original space is linearly bounded by the loss in the intrinsic space. W.l.o.g we may only study the loss in the intrinsic space.

Dear reviewer 7rY6:

Thank you for your reply! We would like to clarify more details in our paper and use better methods to show our experimental results in a clearer way.

1.2 We apologize for any possible lack of clarity in our rebuttal; here are our further response:

1.2.1 Our result does not rely on the strict autoregressive property of time series, as the non-causal relationships could be interpreted as intrinsic noise within our theoretical framework.

1.2.2 We wrote in our rebuttal to clarify that, as you have pointed out, time series may not be strictly autoregressive (in cases like Nebensatz), and our theory is actually describing a similar idea with the 'intrinsic noise' of time series, because of which it would be hard (or even impossible) for models that predict several future frames from several past frames to achieve zero Bayesian Loss.

1.3 We appreciate the feedback on clarity. We did mention the Zip-f assumption in Section 3.2.1 (at the top of page 4), which is within the same section as the other assumptions, though not explicitly listed together: our thought was that we mainly intended to discuss cases where features in intrinsic space follow Zip-f law, which is more of a practical observation compared to previous assumptions; meanwhile other feature degradation patterns may also give similar results (though the formula may be different) w.r.t. impact of horizon. We recognized that we indeed focus on Zip-f case (observed in experiments we conducted), hence we would further clarify it and add it to bullet points in our revised paper.

1.4 Our intention was not to suggest a direct causal relationship between average intrinsic dimension and total intrinsic dimension in the original paper. We agree that this relationship is contingent on specific preconditions cited in the referenced paper. We would like to further clarify this to avoid further misunderstandings in our revised paper.

3.Empirical Validation

We truly appreciate the suggestion of comparing with other possible candidates. For Fig. 1. we have further conducted regression with more possible formula candidates on data points we measured and used AiC, BiC as metrices to compare our theory model with other possible condidates, which better shows the fitness of our proposed theory. We would like to provide further experimental results:

Our theory:

$$f(x)=A+B/x^\alpha$$

Possible candidates could be:

$$g_1(x)=A/x^\alpha$$ $$g_2(x)=A+B\log(x)$$ $$g_3(x)=A\log(x)^2+B\log(x)+C$$

(We observed that $g_3$ could also be a relatively good approximation, but on all curves, $g_3$ would give results with $A>0$, indicating an increase in loss with dataset size (beyond the 'optimal dataset size'), which is not observed in experiments, so it should not be considered a good theory for time series forecasting loss: it is further the case since our theory is either approximately on-pair or better than it.)

Here are the AiC and BiC values for these candidates on our experimental results for different models on different datasets.

ModernTCN:

iTransformer:

MLP/Linear:

On previous tables the best (smallest) one and those with <1 difference in any matrices are marked as bold.

It can be seen that our theory either surpasses the candidates here or is no worse by more than $1$ point. In several cases (e.g., ModernTCN -ETTh2, MLP - Traffic, etc) it beats the 2nd-best one by a large margin. This has better shown the accuracy of our proposed formula.

Thank you for the constructive concerns, and we are looking forward to further discussions!

In our previous reviewer specific rebuttal and in the global rebuttal we have concluded our contribution and novelty, and we would like to further clarify the difference to double descent curve here, for your detailed reference:

2.1.Our claim that longer horizon may lead to worse performance is very different from the double descent curve, which mainly considers the impact of dataset size and model capacity. In fact, in our experiments for look-back horizon in Figure 5, most models (except for linear models whose number of parameters depends largely on look-back horizon) have number of parameters that changes little by look-back horizon, and the decreasing-then-increasing loss is mainly caused by the change in look-back horizon. Meanwhile, as we have mentioned, the time series community has been using 'benefit from longer look-back horizon' as a metric for ‘good models’ for two years[1,2,3,4], so we think that the impact of look-back horizon is not completely understood yet. This impact is actually different from the double descent curve.

2.2. We would like to propose our point of view further that, even if some derived results in some theoretical and experimental work have been understood to some extent, it does not follow that a work providing theoretical and experimental analysis of a phenomenon is meaningless or has little contribution. There are many theoretical analysis papers[10,11,12] proposing a variety of theories to explain the Scaling Law that has been observed [13,14]. We think these works do provide the community with different perspectives on understanding the scaling law, and hence have their novel contributions, even if these proposed theories lead to the well-known scaling law. At the same time, we appreciate the contribution of these works even if some of these theories may not be omnipotent (e.g. cases like overfitting, etc.): these works provide theories that aim not to include every possible scenario, but to focus on the most relevant part of the research problems of Scaling Law, and they have provided novel insight for the community.

Papers cited in the discussion period are presented in our official comment on our global rebuttal, which may act as further reference.

Dear Reviewer oD7J,

Thank you for the detailed and constructive review! We truly appreciate your reviews and questions. Here are our responses:

1. Theory assumptions

Let us give a sketch of our proof, where our assumptions are marked as bold text:

Suppose the true sample is $x[-H:S]$, i.e. we are going to predict $x[0:S]$ from $x[-H:0]$. Now we have a model $m$ that predicts the corresponding intrinsic vector $\phi(x[0:S]) \in \mathcal{M}(S)$ with error bounded by $e_i$, then by assumptions 1, 2 we can verify that the error in the original space is bounded by:

$$\alpha e_i - e - e' \leq L_{ori}\leq \alpha e_i + e + e'$$

Hence, analyzing the loss in the intrinsic space would derive the same result, which justifies our task formulation.

By assumption 3, we may partition the intrinsic space uniformly to get a lower bound of the loss.

We can first decompose the total loss into two components, i.e. Bayesian loss and approximation loss (section 3.2.2). Then the Bayesian loss could be written as:

L_{Bayesian} \leq (1-\eta)K_1^2E[var(P^{-1}[\infty,H]\(x))] + \eta \cdot tr\(\Sigma_S)

Where the first term is a projection term, and the second is a noise term which almost makes no effect on our analysis. By assumptions 4 and 5, the relationship of a sequence with its subsequences is similar to linear projections to the subspace with the minimal eigenvalues if the horizon is large, thus the projection term could be bounded with the eigenvalues of the distribution, ensuring the Bayesian loss is bounded. (section 3.2.3)

Considering the approximation loss. If we have sufficient data, the approximation loss should come from two sources, the intrinsic noise in the data and the effect of unseen dimensions in the intrinsic space. The former is assumed to be uniform and thus can be easily calculated, while the latter should be calculated with help from assumptions 4 and 5, similar to the deduction in Bayesian loss. For the scenario where data is scarce, we don't need the assumptions and the approximation loss basically depends on the distance of a test sample to its nearest training sample in the intrinsic space. (section 3.2.4)

Then we can combine the two loss components and analyze the optimal horizon or other properties. (section 3.3)

2.PCA on intermediate vectors

Thank you for the constructive advice! We obtain intermediate vector representations for multi-variate time series datasets and conduct PCA experiments on them, further validating that these time series datasets do follow a Zip-f distribution with respect to their features. Please refer to Figure 1 in our additional page of PDF uploaded to our “global” rebuttal response, in which we show the results. We observe that such a deep-learning method may face under-training issues, thus causing possible uncertainty in evaluating high-rank features with a limited amount of training data, hence this PCA result could complement each other with the PCA result obtained by directly applying PCA on raw input sequence (under Channel-Independent and RevIN settings) provided in Figure 8 in Appendix G in our original submission.

3.TS DNN models are overoptimized for the common benchmarking datasets.

It is indeed possible that some DNNs have structures overfitting several datasets which may not generalize beyond them. Here we mainly choose models that are designed with simplicity (e.g. linear models, iTransformer models which are basically transformers, ModernTCN models which are composed of time-series-optimized convolutional layers) and the datasets are actually quite different from each other in size (e.g. 1000x difference for data sample number) and in feature degradation (e.g. please refer to PCA results for exchange and other datasets), so we think the power law we observe here should be general, at least to a certain extent. However, we do appreciate the important point you have mentioned. This concern may extend beyond the field of time series forecasting, and we will include a description of it in the next version of our paper.

Dear Reviewer SJaN:

Thank you for the review! We truly appreciate your ideas and questions. Here are our responses:

1.Novelty and contribution related to the 'Main Argument'

1.1. ‘Longer horizon gives worse performance’ is an important fact that could bring insight to the community, and the impact of the look-back horizon is not yet well understood. As presented in the Introduction part in our original submission, in the past two years it has been a common practice for the time series community[1, 2, 3, 4] to try to prove the advantage of the proposed methods by claiming that these methods could benefit from improving look-back horizon; which we show is not necessarily an accurate way to show the superiority of a certain model.

1.2. Our main argument is not only that a longer horizon gives worse performance. Actually, we have provided a theory to explain Scaling Laws in Time Series Forecasting, which pays special attention to the impact of input horizon. This is, actually, very different from proposing only that a long horizon would harm performance, as our theory analyzes the complex impact of different combinations of model width, dataset size and lookback horizon. We also summarize our contribution in the next point we would like to make.

2. Our Theoretical and Experimental Contribution

As stated in our paper, we summarize our contribution as follows:

2.1. We introduce a novel theoretical framework that elucidates scaling behaviors from an intrinsic space perspective, highlighting the critical influence of the look-back horizon on model performance.

2.2. We conduct a comprehensive empirical investigation into the scaling behaviors of dataset size, model size, and look-back horizon across various models and datasets to validate the effectiveness of our claim.

We do agree that proposing novel architectures and approaches achieving state-of-the-art performance is very important for the development of Machine Learning, but we also think that providing insights into important questions, validating hypotheses and proposing related theories could also contribute to the community, which may lead to potential improvement in performance of future models. For example, in our original submission, we have demonstrated in Appendix F why down-sampling can improve performance.

We further clarify our contribution, novelty as well as papers cited in this rebuttal section in the global rebuttal section, which may also act as a reference for your constructive concerns. Please refer to the global rebuttal section for more details.

Dear Reviewer Vwbb,

Thank you for a detailed and inspiring review! We truly appreciate your concerns and adivce! Here are our responses:

1. Fitting experimental results with our derived formula

As displayed in Figure 1,2 and 5 in our original submission, the solid lines are the fitted lines of the theoretical formulas. We recognize that the presentation of these graphs may seem that the lines are obtained by connecting data points, hence we will add further captions to these figures to show that these lines are fitted lines in our next version of paper.

2. With respect to Pretrained Models or Large Mixed Datasets

This is a very insightful question given that there have been works focusing on pretrained time series forecasting models with zero-shot forecasting abilities. However, the theory for scaling laws for pretraining and finetuning still remains a relatively unexplored area compared to the theory for scaling laws for single-dataset models, which is still a topic to study these days. The case of training on Mixed Datasets may be closer to our case which considers single-dataset.

Our theory holds without the need for any modification as long as the dataset itself follows a Zip-f distribution in the intrinsic space. This assumption is sort of natural given that Zip-f law is a natural distribution, whereas we give two analyses from Theoretical and Experimental correspondingly on why it holds for mixed, large datasets.

Theoretically, if a large dataset is composed of $s$l subdatasets of similar size each following the Zip-f law with degradation coefficient $\alpha_1 < \alpha_2 < \ldots < \alpha_s$, each with size $S_i$ and follows Zip-f law: $\lambda_{ij}= A_i/j^{\alpha_i}$, where $\lambda_{ij}$ represents the $j$-th eigenvalue of the $i$-th dataset. Suppose the intrinsic dimensions are orthogonal with each other (hence the PCA components are orthogonal). A simple assumption could be the new intrinsic space is a direct product of the old intrinsic spaces, hence eigenvalues are the union of all the old eigenvalues. After which, an eigenvalue of value $S$ should be the $idx_{total}$-largest eigen value, in which:

$$idx_{total} = \sum_i(idx_{i}) = \sum_i (A_i/S) ^{1/\alpha_i}.$$

When $S$ is small (or correspondingly, when $idx$ is relatively large) this sum is dominated by the small $\alpha$s, and in limitation cases this sum is dominated by the $\alpha_1$ term: $idx \approx (A_1/S)^{1/\alpha_1} + C$, which is approximately a Zip-f distribution.

Experimentally, we use the Mixed Dataset of Traffic, Weather, ETTh1, ETTh2, ETTm1, ETTm2, exchange and ECL to train a Channel-Independent 2-layer 512-dimension MLP and use the Intermediate vector before the decoder layer as a feature vector to do PCA analysis. We found that the result follows a Zip-f distribution for higher-order components (Please refer to Figure 2 in our Extra Page PDF).

(Actually the individual datasets themselves (like Traffic, Weather, etc) are composed of data from different times, areas, etc, so they are actually composed of sub-datasets themselves and they are showing the Zip-f distribution. This also indicates that larger datasets may have similar Zip-f distributions.)

Hence it would not be a bad assumption that Zip-f law still holds for datasets with mixed sub-datasets, and our theory, without major modification, could be applied to these cases.

3.More detailed proof for Section 3.2.2

Yes, we would like to provide a more detailed proof here, which will be updated in the next version of our paper.

$$\begin{aligned} L&=E_{x\sim\mathcal M(H+S)}[(x[H:H+S]-m(x[0:H]))^2]\\ \end{aligned}$$

Let $m^*$ denote the optimal Bayesian model, then it should satisfy:

$$m^*(x[0:H])=E_{x\sim\mathcal M(H+S)}[x[H:H+S]|x[0:H]]$$

Thus:

$$L=E_{x\\sim\\mathcal M(H+S)}[(x[H:H+S]-m^*(x[0:H])+m^*(x[0:H]-m(x[0:H]))^2]\\\\$$ $$L=L_{sum}+L_{cross}, \text{in which}:$$ $$L_{sum} = E_{x\\sim\\mathcal M(H+S)}[(x[H:H+S]-m^*(x[0:H]))^2]+E_{x\\sim\\mathcal M(H)}[(m^*(x[0:H])-m(x[0:H]))^2]$$

is the sum of squares

and

L_{cross} = 2\*E_{x\\sim\\mathcal M\(H+S)} [\(x\[H:H+S]-m^\*\(x[0:H]))\*\(m^*\(x\[0:H])-m\(x\[0:H]))]

is the cross term.

The optimal model (Bayesian model) should satisfy:

$$m^\*(x[0:H]) = \sum\limits_{x[H:H+S]} P(x[H:H+S]\mid x[0:H]) x[H:H+S].$$

Therefore, it follows that: (note the conditional expectation of $x[H:H+S]-m^*(x[0:H])$ given $x[0:H]$ is $0$, which is from the definition of the Bayesian Model)

$$L_{cross}= \\sum\\limits_x P(x[H:H+S]\\mid x[0:H])P(x[0:H])[(x[H:H+S]-m^\*(x[0:H]))*(m^\*(x[0:H])-m(x[0:H]))] =0.$$

Hence, the cross term should be zero, and the loss is a sum of the square terms: one is determined by the capability of the optimal Bayesian model (the Bayesian Error), and the other is about how well the model approximates the Bayesian model (the Approximation Error): $L=L_{cross}+L_{sum}=L_{sum}=L_{Bayesian}+L_{Approx}$.

4.Typos

Thank you for pointing out these typos! We would check typos carefully and further polish our paper.

Thank you for these constructive concerns and suggestion!

We appreciate the feedback on clarity. We would state our assumptions in a clearer and more uniformed way. (Moreover, we would like to propose weaker assumptions which could lead to almost the same result, as proposed in our rebuttal with Reviewer 7rY6. This will be stated clearly in appendix, while current assumptions in our main paper would be further polished for better clarity). The derivation will also be further polished.

As for mixed datasets, we recognize that these scenarios are indeed very important and are rising trends for Time Series Forecasting community. We would state more about mixed datasets, both theoretically and experimentally, in our modified paper; and we hope our work may provide further insight for the community and promote further related researches.

Dear Reviewer urDc,

Thank you for a detailed and inspiring review! We truly appreciate your feedback and concerns! Here are our responses:

1.How the proposed theory extends to multi-variate?

While our deduction is primarily written in a single-variable manner, it can be easily adapted to the multi-variable case. As mentioned in Section 3.2.1, we made the Information-preserving assumption:

There exists a mapping $\phi$ from the original length-$L$ sequence space $\mathcal{O}(L)$ to $\mathcal{M}(L)$, along with an inverse mapping $\phi^{-1}:\mathcal{M}(L)\to \mathcal{O}(L)$ and a constant $e \ll 1$ that is independent of $L$, such that for any $x \in \mathcal{O}(L)$, $\|x - \phi^{-1}(\phi(x))\|_2^2 \leq e$.

For a certain time-series dataset, we assume it has a fixed multi-variable dimension $d_{\text{multi}}$. By substituting $\mathcal{O}(L)$ with $\mathcal{O}(L \times d_{\text{multi}})$, we observe the multi-variable format:

There exists a mapping $\phi$ from the original length-$L$ dimension $d_{\text{multi}}$ sequence space $\mathcal{O}(L \times d_{\text{multi}})$ to $\mathcal{M}(L)$, along with an inverse mapping $\phi^{-1}:\mathcal{M}(L) \to \mathcal{O}(L \times d_{\text{multi}})$ and a constant $e \ll 1$ that is independent of $L$, such that for any $x \in \mathcal{O}(L \times d_{\text{multi}})$, $\|x - \phi^{-1}(\phi(x))\|_2^2 \leq e$.

Another assumption to be checked is the Zip-f assumption on different dimensions of the intrinsic space for the multi-variable case. In Figure 8 in Appendix G in our original submission, the single-variable case is validated by conducting PCA on Channel-Independent time series. We further use the intermediate vector of iTransformer as the feature vector in the intrinsic space of multivariable time series and conduct PCA on them. This can be found in Figure 1 in our additional page of PDF uploaded to our “global” rebuttal response, which validates the Zip-f assumption of the ** Non-Linear Multivariable** case.

Other deductions still hold for multivariate cases (up to a constant factor). Hence our assumptions and deduction can be extended to the multivariate cases.

2. Down-sampling and theory of optimal horizon

Downsampling can be viewed as a projection to a subspace in the intrinsic space and thus has a similar effect as decreasing the horizon. Experimental results (as shown in experiments in previous works like FITS, PatchTST and in our work) show that the projected subspace of higher frequency tends to fall on the large-eigenvalue directions, or the `invisible' dimensions masked by the projection tends to be the more unimportant ones. Although the precise effect of downsampling is unknown or may need further assumptions and methods for precise consideration, it is acceptable that we may approximate the effect of downsampling to be similar to a projection to the first $d_{eff}<d_I(H)$ dimensions in the intrinsic space. The overall loss could be then expressed as:

$$loss_{new} = L(d_{eff})\text{ where $d_{eff} < d_I(H)$.}$$

Hence, if the original intrinsic dimension is larger than (local) optimal $d$, $\partial loss/\partial d>0$, hence reducing $d$ from $d_I(H)$ to $d_{eff}$ would help reduce loss. Otherwise if $d_I(H)$ is already smaller than (local) optimal $d^*$, meaning $\partial loss/\partial d<0$, we would expect no performance improvement from reducing $d$ from $d_I(H)$ to $d_{eff}$.

3. How sensitive the optimal horizon is to changes in data distribution over time? Does the theory accounts for non-stationary time series?

This is indeed a very important question in time series forecasting. In our work we mainly consider the case where training and testing distributions approximately follow the same distribution, because this is a basic case and we can make the gap between training set and test set smaller with normalization methods (like RevIN) or online-training methods. The former methods are included in our theory; however, it is challenging for our theory to model the online-training or test-time-training methods. This is partially because these methods may employ different training and optimization strategies that are significantly different in modeling.

For traditional deep-learning methods (i.e. training on training set, test on test set, no hyper-parameter tuning, etc on test set, do not use timestamp as a parameter of model input, which is the usual case for time series forecasting models like Linear, iTransformer, ModernTCN, etc), if we assume the test set is fixed and we sample training set each time for training. If we assume sampling training set has a distribution difference compared to the test set that has expectation $0$ but a certain variance, then this shift can be modeled into a constant loss term in our theory model, thus making no effect on hyper-parameter choosing (on expectation of training-set sampling).

However, when considering more complex scenarios where distribution differences may be correlated with certain observable parameters (e.g. timestamps), or when online training methods are employed, analyzing the effect of the look-back horizon becomes more intricate. In such cases, both the exact look-back horizon (the length of past data used as input to the current model) and the implied look-back horizon (the length of past data that continues to influence the model during online training) may impact the model's performance. This likely involves a distribution shift in the intrinsic space over time, necessitating distinct considerations for these two horizons. We believe addressing this complexity would be a valuable direction for future research.

4. Others

Thank you for the constructive advice! We would add more detailed description about computational resources and energy consumption in our experiments.

These papers are cited in our dicussions and could possibly act as reference or supplement:

10. U. Sharma and J. Kaplan, “A neural scaling law from the dimension of the data manifold,”, 2020, published on JMLR, 2022.

11.Eric J. Michaud and Ziming Liu and Uzay Girit and Max Tegmark, "The Quantization Model of Neural Scaling", in NeurIPS, 2023.

12.Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma, “Explaining neural scaling laws,” 2021.

13.Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown and Others, "Scaling laws for neural language models", 2020.

14.Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya and Others, "Training Compute-Optimal Large Language Models", 2022.