OLinear: A Linear Model for Time Series Forecasting in Orthogonally Transformed Domain

Thank you for your comments. We sincerely appreciate the reviewer’s recognition of our "well-structured" presentation, the plug-and-play modules that "lower the entry barrier", and an adaptive design that could "inspire further work". Below are our responses:

Q1: On the p-value

• We conduct a significance test (using 7 random seeds) using Student’s t-test between OLinear and iTransformer across various datasets and prediction horizons. As shown below, the better model is in bold when the improvement is statistically significant at the level 0.05 (p-value < 0.05). In other words, OLinear exhibits a clear performance advantage over iTransformer.

• We report the performance with increasing lookback lengths in Appendix I.3. OLinear demonstrates consistent improvements as the lookback horizon increases from 48 to 720, and consistently outperforms state-of-the-art forecasters.

• Regarding robustness across random seeds, please refer to Appendix F, where OLinear exhibits smaller standard deviations and narrower 99% confidence intervals compared to TimeMixer++ and iTransformer.

• We would like to emphasize that MSE and MAE metrics typically degrade in long-term forecasting as the prediction horizon increases. As shown in Table 15 (Page 28), this phenomenon is observed in state-of-the-art time series forecasters, such as TimeMixer++ (ICLR 2025), iTransformer (ICLR 2024), FilterNet (NeurIPS 2024). This phenomenon is reasonable because future series inherently contain greater uncertainty as the horizon extends, much like how predicting the temperature two weeks ahead is less accurate than predicting tomorrow’s temperature.

Q2: On novelty and comparison with SAMformer

• SAMformer (ICML 2024) introduces the Sharpness-Aware Minimization (SAM) mechanism—originally proposed in [1] (ICLR 2021)—into time series forecasting. While channel-wise attention was first introduced by iTransformer, SAMformer adopts the same practice. Its key contribution is a new loss function that smooths the loss landscape of Transformers: $\mathcal{L}^{SAM}\_{train}(\omega ) = \underset{\left \| \epsilon \right \| <\rho }{max}\mathcal{L}\_{train}(\omega +\rho),$ where $\rho >0$ is a hyper-parameter. This can be implemented by the SAM optimizer [1].

• Instead of modifying the loss function, we focus on the architecture design and propose NormLin. Compared with vanilla self-attention, NormLin offers four advantages: lower computational cost, higher-rank attention matrices, smoother gradient flow, and an enlarged optimization space (see Lines 182-198 in the paper). NormLin "can be plugged into existing forecasters to boost performance with minimal engineering effort" (by Reviewer 5fFn). Moreover, our linear-based model, OLinear, outperforms state-of-the-art Transformer-based forecasters—including SAMformer. We will cite and discuss SAMformer in our final draft.

• We compare MSE and standard deviation between OLinear and SAMformer. The SAMformer numbers are taken from its original paper, which uses a look-back window of 512 and thus enjoys extra performance gains. Even under this handicap, OLinear achieves lower MSE and smaller standard deviations in most cases.

• To ensure a fair comparison, we uniformly set the lookback length to 96 and report the following MAEs. The average results are reported across 3 random seeds for SAMformer, using the recommended learning rates and $\rho$ values.
  • On average, OLinear outperforms SAMformer by a substantial margin of 12.1%.
  • Notably, the proposed NormLin layer improves the performance of SAMformer by 10.6% and 5.8% on the Traffic and ECL datasets, respectively.

Q3: On the quantitation of temporally decorrelation

• We report the off-diagonal Frobenius norm of the correlation matrices (size: $96\times 96$) for the original series, and for the series after OrthoTrans or FFT, using a window size of 96. The results below show that OrthoTrans effectively removes temporal correlations and performs better than the FFT operation in decorrelation.

• We also argue that examining correlations in the frequency domain of the feature series after OrthoTrans is of limited relevance: 1) OLinear does not involve any FFT operations. 2) Whether there exists an orthogonal transformation that simultaneously mitigates both temporal- and frequency-domain correlations is a nontrivial question. Further exploration could be a future direction of this work. We will emphasize that the decorrelation is performed temporally in the final draft.

Q4: On quadratic cost of NormLin

• In our view, $\mathcal{O}(N^2)$ is the minimal cost required to model $N$-channel correlations, as even a single linear layer incurs this complexity.

• A straightforward way to reduce this cost is to introduce a bottleneck block: the channels are first projected down to a fixed width (e.g., 64), the NormLin module is applied in the fixed-dimensional space, and then the representation is projected back to $N$ channels. This reduces the computational complexity to $\mathcal{O}(N)$. As shown below, this variant tends to degrade performance, particularly on datasets with a large number of channels, such as ECL and Traffic.

Q5: On hyperparameters

• Because there are substantial differences among datasets, it is common practice (e.g., iTransformer and SAMformer) to fine-tune certain hyperparameters to maximize model performance.

• In the paper, the hyperparameter value ranges are provided in Appendix D. Note that the embedding size $d$ is fixed at 16 for all experiments. We also evaluate the hyperparameter sensitivity of OLinear in Appendix I.6. As shown in Figure 17 and Table 30, OLinear exhibits robust performance across these hyperparameters.

• Furthermore, we freeze a single architecture (learning rate $lr = 5\times10^{-4}$, model dimension $D = 512$, number of blocks $L = 3$, embedding size $d = 16$) and report the average results across four prediction lengths. Compared to the results reported in the paper, the performance degradation is within 1%.

We hope these responses can fully address your concerns. Thank you again for your comments!

Reference

[1] Sharpness-aware minimization for efficiently improving generalization. ICLR 2021.

Thank you for the well-structured rebuttal. Your additional experiments and clarifications resolve most of my earlier concerns. Below are my remarks.

Q1: Significance testing

The inclusion of Student’s t-tests across seven random seeds strengthens the statistical analysis and provides a solid estimate of the robustness of your model.

Q2: Novelty and comparison with existing baselines

The comparison with SAmformer and the reported average 12.1% MAE gain positions OLinear as the stronger baseline. However, what would further elevate this section is a theoretical discussion that contrasts the subspaces produced by OrthoTrans with those implied by a model like SAMformer’s rank-regularised attention. It is still unclear to me whether OrthoTrans provides representational capacity beyond what a simple attention + linear layer equipped with rank regularization, or even LoRA-style methods, can already achieve.

Q3: Quantifying decorrelation

The updated table of off-diagonal Frobenius norms is very helpful. It shows that OrthoTrans efficiently slashes time-domain correlations on every dataset. Given your statement that OrthoTrans is intended only for temporal decorrelation and that extending it to the frequency domain is a separate, non-trivial research question, the current analysis feels sufficient for this paper’s scope. However, a brief sentence in the main text clarifying this design choice would head off potential misunderstandings.

Q4: Quadratic cost

The bottleneck experiment shows the trade-off between complexity and accuracy: reducing the model size lowers the computational cost but also leads to some loss in accuracy on specific datasets. Showing how the performance (MAE or MSE) and latency change as the bottleneck size varies would help readers decide if the full quadratic version of the method is worth it for their own use cases. I suggest adding such an experiment to the paper to make this trade-off more explicit.

Q5: Hyper-parameter robustness

Freezing the learning rate, model dimension, number of blocks, and embedding size while losing $\leq$ 1 % accuracy attests to the method’s stability.

General Comment

The rebuttal effectively strengthens the paper. Further clarifying the main theoretical point raised in Q2 would help improve the overall evaluation.

Point 3: Quadratic cost

• Thanks for your suggestion! We compare the MAE performance and resource footprint across various bottleneck sizes. As shown below, a bottleneck size of 16 appears to be a sweet spot for the performance–resource trade-off. This choice has a performance gap of about 4% compared to the full quadratic version, but with less resource utilization. We will definitely include a new section in the final manuscript to make this trade-off more explicit.

Thanks again for your constructive comments. We are very happy to answer any further questions.

Point 1 (continued): Theoretical discussion of NormLin and self-attention

• To further demonstrate the DoF effects on forecasting performance, we introduce the NormLin_LoRA with the thought of low-rank approximation: $\mathrm{NormLin} _{\mathrm{LoRA} }=\mathrm{Norm} _{\mathrm{L1} }(\mathrm{Softplus} (\mathbf{AB}^{\top} )),$ where $\mathbf{A}, \mathbf{B} \in \mathbb{R}^{N \times d}$. According to the first table, $\mathrm{NormLin} _{\mathrm{LoRA} }$ has fewer DoF than the original NormLin. We report the following MAEs under the architecture of SAMformer. As expected, NormLin outperforms the LoRA-style variants.

• We also report the SVs, ranks, and entropy of the weight matrices for the LoRA variants under the SAMformer architecture. As shown below, the original NormLin exhibits higher SVs, ranks, and entropy than its LoRA counterparts.

• We also compare the NormLin layer with its LoRA variant under the OLinear architecture. As shown below, the reduced DoF of the LoRA variant results in degraded performance.

• We will incorporate this discussion in our final manuscript to further enrich our work.

Point 2: Further clarification on temporal correlation

• Thanks for your suggestion! We will definitely clarify our design of OrthoTrans by emphasizing that the decorrelation is performed temporally in the Introduction and Section 4.2.

We could not be happier to receive such heartwarming remarks. We deeply appreciate that you found our initial responses resolved most of your concerns. Below, we provide our further responses:

Point 1: Theoretical discussion of NormLin and self-attention

• Thanks for your suggestion! We fully agree that further theoretical analysis will elevate our work. We would like to conduct the analysis from the perspective of degrees of freedom (DoF) of the attention matrices. The number of DoF in a matrix/vector is the number of elements needed to completely specify it. The DoF satisfy the following properties [1]:

  • Strictly monotonic functions (e.g., Softplus, exp) that operate element-wise do not impose cross-element constraints, and thus do not change the DoF.
  • For a vector $\mathbf{x} \in \mathbb{R}^N$, its DoF is $N$. When imposing the L1 normalization, i.e., $\mathrm{Norm}_{\mathrm{L1}}(\mathbf{x})$, the DoF is reduced by 1. Similarly, for a matrix $\mathbf{X} \in \mathbb{R}^{N\times N}$, row-wise L1 normalization reduces its DoF from $N^2$ to $N^2 - N$.

• Based on these, we have the following DoF equations for NormLin and self-attention:

• In the above table, we omit scalar parameters for clarity, as they do not affect DoF or ranks. $f(N,d)$ is a piecewise function with a maximum value of $N^2$, defined as $f(N,d)=2Nd-d^2$ if $d \le N$ and $f(N,d)=N^2$ otherwise. The subtraction of $d^2$ in $f(N,d)$ can be understood from the fact that, for any invertible matrix $\mathbf{P} \in \mathbb{R}^{d\times d}$, $\mathbf{Q}'=\mathbf{Q}\mathbf{P}$ and $\mathbf{K}'=\mathbf{K}(\mathbf{P}^{-1})^{\top}$ satisfy $\mathbf{Q}'\mathbf{K}'^{\top}=\mathbf{Q}\mathbf{K}^{\top}$. In SAMformer, the default hidden dimension d is set to 16, which is smaller than the channel count $N$ of most common datasets.

• We can see that the NormLin layer benefits from a larger DoF of the weight matrix (third column) compared to that of the attention matrix in self-attention (last column), since $N^2-N \ge f(N,d)-N$. This indicates a larger search space of attention/weight matrices and potentially better representational capacity [2].

• Empirically, we report the minimum singular values (SVs), median SVs, ranks, and entropies of the attention and weight matrices for SAMformer and SAMformer+NormLin (denoted as NormLin in the following table). The lookback and prediction lengths are set to 96. Since the attention matrix in SAMformer is input-dependent, we report the mean, minimum, and maximum values (denoted as S_mean, S_min, S_max, respectively) across the test sets for SAMformer. The matrix rank is computed as the number of SVs greater than $10^{-3}$. The SAM optimizer is used with the recommended hyperparameters in the SAMformer paper.

• We can observe that, compared with the vanilla SAMformer, the plug-in NormLin layer generally produces larger SVs, higher ranks, and, at the same time, better prevents attention entropy collapse. These results indicate better representational capacity than simple attention with rank regularization. In addition, this rank improvement phenomenon is consistent with the findings in Appendix E.2 of our paper.

References:

[1] Strang, Linear Algebra and Its Applications, International Thomson Publishing, 3rd edition.

[2] Attention is not all you need: pure attention loses rank doubly exponentially with depth. ICML 2021.

Thank you for your timely response and raising your score : ) We are glad that all your initial concerns have been effectively addressed. Your comments are important to enhance the quality of our manuscript and we will definitely incorporating these revisions in our camera-ready version - thank you so much!

(b) The weight matrix is trained to approximate the correlation matrix among channels $\mathrm{CorrMat}_v$.

• In Figure 5 of the paper, we observe that the learned NormLin weights exhibit strong similarity to $\mathrm{Softmax}(\mathrm{CorrMat}_v)$. Motivated by this observation, we pre-compute the correlation matrix among channels $\mathrm{CorrMat}_v$ with the trainset and directly use $\mathrm{Softmax}(\mathrm{CorrMat}_v)$ as a fixed weight matrix, resulting in a variant named $\mathrm{NormLin}_c$:

We refer to OLinear with this variant as OLinear-C.

• We then compare the average performance of OLinear-C with OLinear across multiple prediction lengths. ETT denotes the average results across the ETTh1, ETTh2, ETTm1 and ETTm2 datasets. As shown below, the average performance gap between OLinear and OLinear-C on these datasets is negligible (only 0.2%)!

• If the ideal weight matrix is the matrix $\mathrm{Softmax} (\mathrm{CorrMat} _v)$, this partly explains the performance advantage of NormLin over self-attention.

  • Because attention matrices in self-attention are input-dependent, it is harder for them to converge to the input-independent multivariate correlation matrix.
  • In contrast, NormLin uses an input-independent weight matrix, $\mathrm{Norm} _{\mathrm{L1} }(\mathrm{Softplus} (\mathbf{W} ))$, making this approximation easier.

• The matrix $\mathrm{Softmax} (\mathrm{CorrMat} _v)$ is usually high-rank. How does NormLin_LoRA, whose weight matrix is $\mathrm{Norm} _{\mathrm{L1} }(\mathrm{Softplus} (\mathbf{AB}^{\top} ))$ also perform well, given that $\mathbf{AB}^{\top}$ has a low rank of $d$? The reason is that $\mathrm{Norm} _{\mathrm{L1} }(\mathrm{Softplus} ( \cdot))$ rescales and rotates each row differently and thus could increase the matrix rank [1], enabling a better approximation of $\mathrm{Softmax} (\mathrm{CorrMat} _v)$.

• We will accordingly tone down the analysis of weight-matrix rank (Lines 187-193 ) and DoF (Lines 196-198) in the final paragraph of Section 4.3, and relocate the OLinear-C exposition from Section 5.2 to Section 4.3, to better clarify the mechanism underlying the NormLin layer.

Quadratic cost

Thanks for your suggestion! We will definitely add a figure showing performance versus bottleneck size in the relevant section to improve readability.

We are eager to hear your feedback. We sincerely hope that our new comments can address your primary remaining concerns. We’d deeply appreciate it if you could let us know whether your concerns have been addressed -- thank you so much!

Thanks for your time,

Authors of #10716

References:

[1] DenseHMM: Learning Hidden Markov Models by Learning Dense Representations. NeurIPS 2020.

[2] Feature selection, L1 vs. L2 regularization, and rotational invariance. ICML 2004.

Dear Reviewer jUXb,

We highly appreciate your timely and constructive responses, and we are glad that you found our expanded discussion around Degrees-of-Freedom very useful, and the quadratic-cost analysis clearly illustrates the efficiency trade-offs.

Yes, we agree with the reviewer that, from our LoRA experiments, the observed performance gains of NormLin do not primarily arise from increased rank or Degrees-of-Freedom alone. To further understand the method’s success reasons, we added more experiments and in-depth analysis, showing that: (a) The L1 normalization contributes more to NormLin's performance gains than the Softplus function, and (b) The weight matrix is trained to approximate the correlation matrix among channels $\mathrm{CorrMat}_v$.

(a) The L1 normalization contributes more to NormLin's performance gains than the Softplus function.

What component affects NormLin's performance most?

• NormLin consists of two components:

  • The transformation function (Softplus);
  • Normalization strategy (L1 norm).

• We add ablation studies on 5 transformation functions (Softplus, Identity, Softmax, Sigmoid, ReLU) and 3 normalization strategies (L1, L2, None), where Identity denotes no transformation and None denotes no normalization. The average results across four prediction lengths are presented below.

• From above, we found that the L1 normalization contributes more to NormLin's performance gains than the Softplus function. Specifically,

  • (i) the L1 norm outperforms L2 norm and no normalization by 10% and 12%, respectively;
  • (ii) under L1 normalization, Softplus outperforms Identity, Softmax, Sigmoid, and ReLU by 5%, 1%, 0.2%, and 0.3%, respectively.

• The reasons behind the performance advantage of L1 norm could be intricate. It has become a natural choice since it induces a probabilistic distribution over each row of the weight matrix.

  • Without L1 normalization, each row $\mathbf{W}_{i,:}$ effectively includes a learnable scaling parameter $c_i$, whether using L2 normalization or no normalization at all.. However, this scaling is rendered meaningless by the subsequent LayerNorm operation, which eliminates any learned magnitude differences.
  • In contrast, L1 normalization emphasizes the relative distribution of weights within each row rather than their absolute scale, which may lead to improved performance.

Thank you for your positive comments on our work. We sincerely appreciate the reviewer’s recognition of our work as theoretically well-founded, experimentally comprehensive, and achieving state-of-the-art performance. Here are our responses to your concerns:

W1: On the deployment cost

• We report the training and inference resource footprint of OLinear and baseline forecasters in Table 37 (Page 48), including inference time and GPU memory consumption. OLinear achieves higher inference efficiency than state-of-the-art Transformer-based forecasters (e.g., TimeMixer++, iTransformer, and CARD), benefiting from the simplified NormLin module. Moreover, Table 38 shows the efficiency differences before and after applying the NormLin module to Transformer-based forecasters.

W2: On more baselines

• We compare OLinear with RLinear and TSMixer using the MSE metric (see below). The results of RLinear and TSMixer are taken from iTransformer and SAMformer, respectively. On average, OLinear outperforms these two forecasters by 17.3% and 25.6%, respectively.

• In Section 5.1, OLinear is compared with 11 carefully selected and widely acknowledged state-of-the-art forecasting models. We believe these results demonstrate the performance superiority of OLinear. For completeness, we will also include discussions on RLinear, TSMixer, StemGNN, and DEPTS in the final draft.

W3: On the claim of “universal token dependency learner”

• On Line 289, we explicitly restrict the claim “universal token dependency learner” to the field of “time series forecasting.” However, on Lines 278 and 312, this field restriction was unintentionally omitted. We will revise these two sentences in the final draft to ensure accuracy. Thank you for pointing this out!

Q1: On the gradient dynamics

• In Appendix B, we analyze the Jacobian matrix of the non-linear transformations in the self-attention mechanism and the NormLin layer. For self-attention, we have $\frac{\partial \mathbf{c} }{\partial \mathbf{a} } = \mathrm{Diag} (\mathbf{c}) - \mathbf{c}\mathbf{c}^{\mathsf{T}},$ where $\mathbf{c} \triangleq \mathrm{Softmax}(\mathbf{a}) \in \mathbb{R}^N$, and $\mathrm{Diag} (\mathbf{c})$ is the diagonal matrix with $\mathbf{c}$ as its diagonal.

• For the NormLin layer, it holds that

Thank you for your invaluable review. We sincerely appreciate the reviewer’s recognition that our plug-and-play OrthoTrans and NormLin modules are "really interesting" and the experiments are "exhaustive" and "extensive". Below are our responses to the concerns:

Q1: On the computation of $\mathrm{CorrMat}\_{t}$ and 'asymmetric' normalization

• We first clarify our process of computing the temporal correlation matrix $\mathrm{CorrMat}\_{t}$. For clarity, we use the univariate series. Let $\mathbf{x}^{\text{train}} \in \mathbb{R}^{M}$ denote the training series. We then generate $T$ lagged sub-series (whose length is $M-T+1$): $$ \mathbf{s}_{i} = \mathbf{x}[i : M - T + i],\quad i = 0, 1, \dots, T-1.

We would like to thank you again for your constructive comments!

• To fully address your concerns about the quadratic cost of the NormLin layer, we introduce a bottleneck block to reduce computational complexity:

  • the channels are first projected to a fixed width (denoted as bottleneck size),
  • the NormLin module is applied in this lower-dimensional space,
  • and the representation is finally projected back to $N$ channels.

• In this way, the complexity of the NormLin layer is reduced from $\mathcal{O}(N^2)$ to $\mathcal{O}(N)$. We report the MAEs and resource footprints under various bottleneck sizes below.

• From above, we find that:

  • (1) the bottlenecked NormLin variants incur approximately a 4% performance degradation,
  • but (2) consume fewer resources, especially on datasets with a large number of channels.

• This provides readers with additional flexibility to decide whether the full quadratic version of NormLin is worthwhile for their specific use cases.

We sincerely hope that our new comments can address your concerns. We’d deeply appreciate it if you could let us know whether your concerns have been addressed.

Thanks for your time,

Authors

We would like to thank you again for your constructive comments! To fully address your concerns, we compare OLinear with additional lightweight baselines, including several recently accepted works: SimpleTM (ICLR 2025), TQNet (ICML 2025), TimePro (ICML 2025), and TimeBase (ICML 2025). We report the MAEs below, highlighting the best results in bold. The lookback length is uniformly set to 96. Remarkably, OLinear outperforms SimpleTM, TQNet, TimePro, and TimeBase by 5%, 5%, 7%, and 16% on average, respectively, establishing OLinear as a strong new baseline for lightweight time series forecasting.

Thanks again for your constructive comments. We are very happy to answer any further questions.

Dear Reviewer CEMa,

We highly appreciate your constructive comments, and we are glad that you found our initial responses have addressed your concerns on recent lightweight baselines, quadratic computational complexity and implementation details. We address your questions point-to-point in the following.

Q1 (Part 1): Can you show that $\mathrm{CovMat} (\mathbf{Q}_i^T\mathbf{x})$ is actually diagonal?

• Thank you for raising this point. Theoretically, suppose the temporal correlation matrix $\mathrm{CovMat}_t$ admits the eigendecomposition $\mathrm{CovMat}_t = \mathrm{Q}_i \mathbf{\Lambda} \mathrm{Q}_i^T$, where $\mathrm{Q}_i$ is an orthogonal matrix and $\mathbf{\Lambda}$ is a diagonal matrix. If the vector $\mathbf{x}$ is normalized (zero mean and unit variance), then the following holds:

$\mathrm{CovMat} (\mathbf{Q}_i^T\mathbf{x})=\mathbb{E}[\mathbf{Q}_i^T \mathbf{x} \mathbf{x}^T \mathbf{Q}_i]=\mathbf{Q}_i^T \mathbb{E}[\mathbf{x} \mathbf{x}^T] \mathbf{Q}_i=\mathbf{Q}_i^T \mathrm{CorrMat}_t \mathbf{Q}_i = \mathbf{Q}_i^T \mathbf{Q}_i \mathbf{\Lambda} \mathbf{Q}_i^T \mathbf{Q}_i =\mathbf{\Lambda},$

which is a diagonal matrix.

• Empirically, we quantify the effectiveness of OrthoTrans by reporting the Frobenius norm of the off-diagonal elements in the correlation matrices for the original series, and for the series transformed by OrthoTrans, DFT, or Haar wavelets, using a window size of 96. For DFT, correlations are computed using only the real part of the transformed signal, while for Haar wavelets, only the high-frequency components are used. The results below show that OrthoTrans effectively mitigates temporal correlations and outperforms both DFT and Haar wavelets in decorrelation on both the training and test sets.

Q1 (Part 2): On global mean (std) and local mean (std)

• We would like to show that the local mean and standard deviation (std) of the sub-series are very close to the global mean and std, since each sub-series contains only $T-1$ fewer elements than the global series (of length $M \gg T$). We report the following metrics: $\mu \_{err}=\max_{i} \left | \frac{\mathrm{mean} (\mathbf{s} \_i)}{\mu}-1 \right |$, and $\mathrm{std} \_{err}=\max_{i} \left | \frac{\mathrm{std} (\mathbf{s} _i)}{\mathrm{STD} }-1 \right |$, where $\mu$ and $\mathrm{STD}$ denote the global mean and std, respectively.

Q1 (Part 3): On classic autocorrelation estimation (ACF)

• We respectfully argue that for a univariate series, computing the ACF inherently involves comparing two subseries of the original signal, as it measures the similarity between the signal and its lagged versions.

• Assume that the series $\mathbf{x} \in \mathbb{R}^M$ has been normalized (i.e., zero mean and unit variance). The ACF can be regarded as the covariance between two sub-series: $\gamma \_k = \mathrm{E} \left [ \mathbf{x} \_t \mathbf{x} \_{t+k} \right ]=\frac{1}{M-k} \sum_{i=0}^{M-k-1} \mathbf{x}\_i \mathbf{x}\_{i+k} = \mathrm{Cov}(\mathbf{s}\_{1,k}',\mathbf{s}\_{2,k}' ),$

where $\mathbf{s}\_{1,k}'=\left [ \mathbf{x}\_0, \mathbf{x}\_1, \cdots,\mathbf{x}\_{M-k-1} \right ]$, and $\mathbf{s}\_{2,k}'=\left [ \mathbf{x}\_k, \mathbf{x}\_{k+1}, \cdots,\mathbf{x}\_{M-1} \right ]$. We assume that the means of $\mathbf{s}\_{1,k}'$ and $\mathbf{s}\_{2,k}'$ are 0. Both $\mathbf{s}\_{1,k}'$ and $\mathbf{s}\_{2,k}'$ have length $M-k$.

• In OrthoTrans, $\gamma \_k = \mathrm{Cov} (\mathbf{s}\_i, \mathbf{s}\_{i+k} ),$ where $\mathbf{s}\_{i}=\left [ \mathbf{x}\_i, \mathbf{x}\_{i+1}, \cdots,\mathbf{x}\_{M-T+i} \right ]$ and $\mathbf{s}\_{i+k}=\left [ \mathbf{x}\_{i+k}, \mathbf{x}\_{i+k+1}, \cdots,\mathbf{x}\_{M-T+i+k} \right ]$.

• By comparing $\mathbf{s}\_{1,k}'$, $\mathbf{s}\_{2,k}'$ with $\mathbf{s}\_{i}$, $\mathbf{s}\_{i+k}$, we can observe that $\mathbf{s}\_{i}$ and $\mathbf{s}\_{i+k}$ are the truncated versions of $\mathbf{s}\_{1,k}'$ and $\mathbf{s}\_{2,k}'$ (with $T-k-1$ fewer entries) , respectively. In other words, our sliding window approach is a (slightly) truncated version of the classical ACF.

Q1 (Part 4): Performance comparison of OrthoTrans and ACF

• We compute $\gamma \_k = \mathrm{E} \left [ \mathbf{x} \_t \mathbf{x} \_{t+k} \right ], k=1,2,\cdots, T-1$, and obtain the vector $[1, \gamma \_1, \gamma \_2, \cdots, \gamma \_{T-1}]$. Based on this vector, we construct the symmetric Toeplitz correlation matrix and compute the corresponding Q matrix. We denote this method as ACF.

• We directly compare the differences between the correlation matrices and $\mathbf{Q}$ matrices obtained from OrthoTrans and ACF using the metric $\mathrm{Diff}_{\mathrm{F\text{-}norm}}(\mathbf{A},\mathbf{B} ) = \left \|| \mathbf{A}-\mathbf{B} \right \||_F / T^2$, where $\left \|| \cdot \right \||_F$ denotes the Frobenius norm. As shown below, the differences are negligible.

• We report the off-diagonal Frobenius norm of the correlation matrices (size: $96 \times 96$) for the original series, and for the series transformed by OrthoTrans and ACF. The following results show that OrthoTrans and ACF achieve comparable and strong decorrelation performance on both training and test sets.

• We further compare the forecasting performance of OrthoTrans and ACF. As shown below, OrthoTrans performs comparably to ACF.

Q2 (Part 1): On OrthoTrans and PCA

• Thank you for your clear suggestion! We refer to the suggested variant as PCA. It first segments the series $\mathbf{x}\in \mathbb{R}^{M}$ into $N'$ non-overlapping windows of length $T$, forming the matrix $\mathbf{W}\in \mathbb{R}^{N' \times T}$. Then the correlations are computed among the columns of $\mathbf{W}$.

• In essence, OrthoTrans and the PCA variant differ in their unfolding strategies: although both use a patch size of $T$, OrthoTrans employs sliding windows with stride 1, whereas PCA uses a non-overlapping segmentation with stride $T$.

• We further introduce a variant, OrthoTrans-C, for comparison:

  • In OrthoTrans-C, we do not use the CA strategy. Given the multivariate training series $\mathbf{X}^{train} \in \mathbb{R}^{N \times M}$, we apply PyTorch’s unfold function along the time dimension (second dimension) with patch size $T$ and stride 1, resulting in the matrix $\mathbf{X}^{train}\_{fold} \in \mathbb{R}^{N \times M' \times T}$. Then we flatten the first two dimensions of $\mathbf{X}^{train}\_{fold}$ to obtain a matrix of size $(N \cdot M') \times T$, and compute the correlation matrix across its columns.

Q2 (Part 2): Decorrelation performance of OrthoTrans, PCA and OrthoTrans-C

• We compare the decorrelation effectiveness of OrthoTrans, PCA and OrthoTrans-C using the off-diagonal Frobenius norm with a window size of 96. In general, OrthoTrans and OrthoTrans-C achieve better decorrelation performance than PCA.

• To further study the relationship between decorrelation performance and unfolding stride in PCA, we report the following results on the testsets of different datasets, which clearly show that smaller strides lead to better decorrelation performance.

Q2 (Part 3): Forecasting performance of OrthoTrans, PCA and OrthoTrans-C

• We apply these decorrelation methods to OLinear and observe that they achieve comparable performance. The results also verify the robustness of OLinear to different $\mathbf{Q}$ matrices.

• We also examine the robustness of OrthoTrans, PCA, and OrthoTrans-C when the $\mathbf{Q}$ matrices are computed using only the first 10% of the training set. We observe that OLinear performs robustly in this setting, consistent with the finding in Appendix I.8.

• The newly added experiments show that OrthoTrans, ACF, PCA, and OrthoTrans-C perform similarly when applied to OLinear, demonstrating its robustness to different decorrelation methods. We will include a discussion of these variants in a new appendix section and incorporate the results presented here to further strengthen our work. Thank you again for your constructive comments!

Q3-6: On the dimension extension, temporal NormLin, lightweight baselines, and computational complexity

• We appreciate the reviewer’s constructive comments and are glad that our responses have addressed the concerns.

We are eager to hear your feedback. We sincerely hope that our new comments can address your remaining concerns. We’d deeply appreciate it if you could let us know whether your concerns have been addressed. Thank you so much!

Thanks for your time,

Authors of #10716

Thank you for your detailed response. You have treated most of my concerns, so I have decided to raise my score. However, I still would raise some important issues that I would like to ask the authors to solve.

• Please be more mathematically rigorous. If I understand correctly, you assume that $CorrMat_t$ is close to $CovMat_t$ ( what is $CorrMat_t$ ? Is it the correlation matrix for a fixed channel?). In this case, you need to use $\approx$ in all transitions like "$CorrMat_t = Q_i \Lambda Q_i^\top$".
• If ACF and your strategy give the same results, then I would suggest the authors to stick to ACF, because it is more intuitive and familiar to the broad reader. To me, it was very confusing to follow the paper, especially with $CovMat_t$ notation (by the way, it is confusing to write $CovMat_t$ and call it "the temporal correlation matrix").
• When you compute the Frobenius norm of the off-diagonal elements, the values are not that meaningful as it depends on the size of the matrix. Maybe dividing by the number of off-diagonal elements would be more intuitive in order to have a value within [0,1].

Please incorporate all the modifications we have discussed during the rebuttal. This will greatly improve the quality of the paper.

Best regards, Reviewer CEMa

Dear Reviewer CEMa,

We highly appreciate your constructive responses and great efforts in our multi-round discussion. We are encouraged that you found our responses have addressed most of your concerns! We would like to respond to your remaining questions as follows.

Q1: If I understand correctly, you assume that $\mathrm{CorrMat}_t$ is close to $\mathrm{CovMat}_t$ ( what is $\mathrm{CorrMat}_t$? Is it the correlation matrix for a fixed channel?). In this case, you need to use $\approx$ in all transitions like "$\mathrm{CorrMat}_t=\mathbf{Q} _i \mathbf{\Lambda} \mathbf{Q} _i^T$".