MixLinear: Extreme Low Resource Multivariate Time Series Forecasting with $0.1k$ Parameters

Comment1:
In its current form, the paper only reports optimal forecasting but does not consider a distribution across multiple seeds. Is MixLinear consistent in achieving high predictive performance? How large is the spread compared to other approaches?

Response:
Thank you for your comment. To evaluate the stability, robustness, and predictive performance of the MixLinear model across various datasets and forecasting horizons, we conducted five independent runs with different random seeds. The table highlights the model's sensitivity to random initialization and its ability to deliver consistent performance by reporting the corresponding mean and standard deviation.

We have added the new results into Appendix B.5 Table 10 Page 18 of the revised manuscript.

Comment2:
Can you comment on how MixLinear's hyperparameters can be effectively found, e.g., cutoff values?

Response:
To evaluate the effect of the cutoff frequency used by the low-pass filter, we conducted new experiments by varying the LPF cutoff frequency threshold from $n^{\text{LPF}} = 1$ to $n^{\text{LPF}} = 19$ and measured MixLinear's prediction performance. The results show that the prediction performance of MixLinear decreases significantly as the LPF threshold decreases from $n^{\text{LPF}} = 5$ to $n^{\text{LPF}} = 1$.

To balance performance and computational efficiency, a cutoff frequency of $n^{\text{LPF}} = 5$ is generally optimal for resource-constrained environments.

We have added these results to Appendix B.2, Table 7 (Page 16) of the revised manuscript.

Comment3:
The manuscript only considers time series forecasting problems with strong seasonality. It seems the approach would not work well for problems where this is not applicable. It seems meaningful to clearly spell out this assumption.

Response:
Thank you for your comment. We have added the Exchange dataset, which does not exhibit strong seasonality. Despite the lack of seasonal patterns, MixLinear demonstrates better predictive performance on this dataset, showcasing its versatility across diverse time series forecasting scenarios.

Additionally, we have added evaluations on the Solar dataset to further assess MixLinear's performance. The new results have been included in Section 4.2, Table 1 (Page 6) of the revised manuscript.

Comment4:
You claim that "Transformer-based models offer high forecasting accuracy, but they are often too compute-intensive to be deployed on devices with hardware constraints [...]". This claim is not substantiated in light of various pruning techniques, knowledge distillation, or neuron transplantation. Could you provide proof? Your paper currently only considers the training phase, not the inference setup.

Response:
Thank you for your comment. We have presented the inference time for MixLinear and other models. The results show that MixLinear achieves superior performance while being the most lightweight model, with the least inference time among all methods evaluated.

These findings highlight MixLinear's suitability for deployment on resource-constrained devices, even when compared to Transformer-based models optimized through techniques like pruning or distillation. We have added these results to Section 4.3, Table 2 (Page 7) of the revised manuscript.

Comment5:
"By decoupling channel and periodic information from the trend components [...]" - there are several prior works that attempt to achieve this. How does your approach compare to these or other works? Especially with regard to drastically reducing computational complexity by computing mostly in the frequency domain, achieving a time complexity of $O(L \log L)$, where $L$ is the sequence length.

Response:
Thank you for your comment and for highlighting these related works.

• ReCycle employs primary cycle compression (similar to SparseTSF) to reduce the computational complexity of attention mechanisms in long time series. Our approach operates on the downsampled trend using linear layers and further enhances efficiency by leveraging both the time and frequency domains. This results in MixLinear (0.195K parameters) being significantly more lightweight than SparseTSF (0.92K parameters).

• Profile Neural Networks utilize convolutional layers to model time series. In contrast, our method relies on simple linear layers, which are computationally efficient while capturing both temporal and spectral patterns.

• FourCastNet applies Fourier transformations with a Vision Transformer backbone. Unlike FourCastNet, MixLinear employs linear layers in both domains to achieve superior efficiency with a time complexity of $O(n \log n)$, where $n = L / w$ and $w$ is the period of the sequence.

We have added this analysis to Appendix B.6 (Page 19) of the revised manuscript.

Comment11:
Is it meaningful to consistently use the Landau-/Big-O notation when assessing the algorithm's complexities, e.g., p. 4 in the subsection "Segment Transformation"?

Response:
Thank you for your comment. We appreciate your suggestion regarding the consistent use of Landau/Big-O notation for assessing algorithmic complexities. We continue to use Landau/Big-O notation to ensure that all complexity analyses are consistently expressed for clarity and precision. This standardizes the discussion and provides a clear comparison of computational efficiency across different methods.

Comment12:
Have the authors considered working on the differences between time steps or the residuals of the time series and a simple smoothing average curve? A vast number of publications have identified this to be beneficial for predictive performance, training convergence, and reduced parameter count.

Response:
Thank you for your comment. We use 1D CNN aggregation to smooth the curve and incorporate the residual of the original series to preserve information while generating the trend. We have added discussions to Appendix A.1 and Algorithm 1 (lines 1-3) on Page 13 of the revised manuscript to clarify this.

Comment13:
It seems meaningful that the sign of "Improvement" in Table 1 should be inverted since lower MSE indicates a better model. Furthermore, since it is not consistently an improvement, does the term "Difference to best baseline" or similar make more sense?

Response:
Thank you for your suggestion. We have changed the term "Improvement" in Table 1 to "Difference." This change better reflects the nature of the metric, particularly considering that a lower MSE signifies superior performance.

Comment14:
Can the authors comment on the suggested method's capability to capture peaks or other extremes of the time series well?

Response:
Thank you for your comment. MixLinear is designed to effectively capture peaks and extremes in time series data by leveraging its dual-domain approach:

• Time Domain: The model captures localized variations and dependencies through intra-segment and inter-segment transformations, enabling it to respond to abrupt changes or spikes.
• Frequency Domain: MixLinear uses spectral analysis to identify and reconstruct critical patterns, including those associated with peaks and extremes.

This complementary integration of temporal and frequency domain features ensures that MixLinear can model both subtle trends and sharp variations in the data.

Comment15:
Are you willing to release the source code of your method and its evaluation for the sake of reproducibility?

Response:
Thank you for your comment. We are committed to ensuring the reproducibility of our research and will release the source code along with the evaluation details to facilitate further exploration and validation by the community.

Comment6:
The manuscript claims, "The high computational demands and large memory requirements of these models hinder their deployment for LTSF tasks on resource-constrained devices". However, it is unclear how much effort went into optimizing these models. If only MixLinear has been highly optimized, the empirical evaluation may favor MixLinear.

Response:
Thank you for your comment. To ensure a fair comparison, we adopted the optimized hyperparameters provided in the official GitHub repositories of all baseline models. All datasets used are benchmark datasets, and baseline models have been optimized by their respective developers on these datasets.

Comment7:
Carefully check the paper for grammatical errors as well as precise phrasing. Examples include:

• "On the other hand, the linear models aim [...]" - the second "the" is superfluous.
• "Overhead by employing either decomposition methods [...]" - what does "either" refer to here?
• "[...] by modeling intra-segment and inter-segment variations [...]" - "segment" is unclear. Could you clarify this term?

Response:
Thank you for your feedback. We have addressed these issues in the Abstract of the revised manuscript. "Segment" refers to a fragment of the time series data.

Comment8:
Fig. 1 shows an example of the Electricity dataset, but:

1. The dataset has not been introduced at this point.
2. It does not indicate if and how data has been normalized, making MSE values hard to estimate.

Response:
Thank you for your comment. We have introduced the Electricity dataset before the figure in the Introduction section. Regarding normalization, all models (including MixLinear and baselines) use the same normalization method (StandardScaler). This ensures consistency across comparisons. Details on normalization have been added to Appendix A.4 (Page 15) of the revised manuscript.

Comment9:
Please improve Fig. 2 by:

• Using correct mathematical symbols, e.g., for the set of real numbers.
• Increasing font size.
• Defining all mathematical symbols (e.g., LPF) in the caption.
• Ensuring consistent reading direction.

Response:
Thank you for your feedback. We have revised Figure 2 (Page 4) of the revised manuscript to address these issues.

Comment10:
How does the frequency domain part of MixLinear differ from Fourier Neural Operators (FNO)?

Response:
Thank you for your comment. FNO primarily focuses on learning resolution-invariant solution operators for parametric PDEs, such as the Navier-Stokes equations. In contrast, MixLinear leverages the frequency domain to capture spectral features like phase and amplitude patterns, specifically for time series forecasting. Unlike FNO, which is tailored for PDEs, MixLinear integrates spectral analysis to model temporal dependencies effectively while remaining computationally efficient.

Comment1:

The performance improvements seem to be marginal. Specifically, except for the ETTh1 dataset, the performance on other datasets is not good enough compared to SparseTSF and FITS.

Response:
Thank you for your comment. Our key contribution lies in significantly reducing the model size while maintaining competitive prediction performance. Many existing models are too large to be deployed on devices with limited hardware resources. In contrast, our model is designed to be lightweight and suitable for such resource-constrained devices, offering efficient operations without significantly compromising prediction accuracy.

We have incorporated two additional datasets (Exchange and Solar) and two new baselines (iTransformer and SCINet) into our experiments, which are reflected in Table 1 in Section 4.2 (Page 6) of the revised manuscript. The results demonstrate that MixLinear continues to achieve better or comparable prediction performance compared to other baselines.

Comment2:

The experimental section of this paper lacks sufficient visualization of experimental results. For instance, what are the training and convergence results of the model compared to other linear models? What is the predictive performance of the model?

Response:
Thank you for your comment. We have added new figures to visualize our experimental results. Specifically:

• We visualize the training convergence of MixLinear under the input-720-predict-720 setting on the Exchange dataset, comparing it to SparseTSF, FITS, DLinear, PatchTST, and iTransformer. As shown in Figure 3 (Page 20), MixLinear demonstrates faster and smoother convergence compared to other models.
• We present the predictive performance of MixLinear compared to other baseline models under the input-720-predict-96 and input-720-predict-192 settings on the Exchange dataset. Figures 4 (Page 21) and 5 (Page 22) show that MixLinear consistently outperforms all baselines.

This analysis has been added to Appendix B.7 and B.8 of the revised manuscript.

Comment3:

The paper lacks a hyperparameter (such as $w$) search experiment, and the authors did not provide parameter information corresponding to the optimal results across all datasets (only provided for Electricity). Is the model's performance sensitive to $w$ across different datasets?

Response:
Thank you for your feedback. To evaluate the effect of the period $w$ used for downsampling, we conducted new experiments by varying $w$ from 2 to 36 across forecasting horizons of 96, 192, 336, and 720. We added the results to Appendix B.5, Table 10 (Page 18) of the revised manuscript.

For most datasets, $w = 24$ achieves the best performance. However, on the Exchange dataset (financial data that lacks strong seasonality), smaller values of $w$ (e.g., $w = 2$) yield better results, suggesting a different temporal structure where shorter periods are more suitable.

Comment4:

In Appendix Table 7, the authors mentioned filters with $n = 15$ and $n = 19$ points, but the length of the corresponding trend sequence is only 30 (for ETTh1). The FFT of the actual sequence should be conjugate symmetric, meaning it contains only 16 points of valid information. A 19-point filter introduces redundancy and high-frequency noise.

Response:
Thank you for your feedback. We performed additional experiments with MixLinear using different low-pass filter (LPF) values. While an LPF greater than 15 may introduce redundancy and high-frequency noise due to components from negative frequencies, our experiments show that in certain cases, an LPF greater than 15 can still yield better performance.

The new results have been added to Appendix B.2, Table 7 (Page 16) of the revised manuscript.

Comment5:

Another concern is about the frequency domain transformation. Temporal trend decomposition already removes the inherent periodic and detailed information in the data, emphasizing the overall structure. Additionally, performing frequency domain transformations on sequences after periodic downsampling (where $w$ is often large, e.g., 24) may cause severe aliasing effects for most time series. The authors should provide a theoretical analysis or empirical demonstration of how their method addresses or mitigates these aliasing effects.

Response:
Thank you for your comment. While periodic downsampling with a large $w$ (e.g., 24) can lead to severe aliasing effects, our approach mitigates this issue through an aggregation step performed before downsampling. Specifically:

• We apply a 1D convolution with kernel size $w$, aggregating all information within each period at every time step.
• After aggregation, we downsample the series by the period $w$, resulting in the trend component.

This approach effectively decouples the periodic and trend components, producing a compact representation where each trend time point encapsulates the complete information from one period. This method significantly reduces the risk of aliasing while preserving essential characteristics of the time series.

Comment6:

Although the authors performed ablation studies, they did not explicitly explain the role of key components in the model. For example, what kind of information is learned by the temporal and frequency domain methods? In the temporal transformation, the segment transformation used by the authors is essentially similar to the operation of an MLP-Mixer, which mixes data along two directions. The interpretability of this operation is not clear.

Response:
Thank you for your feedback. In the time domain, MixLinear captures intra-segment and inter-segment variations by decoupling channel and periodic information from trend components, breaking the trend into smaller, manageable segments. In the frequency domain, the model captures variations by mapping these decoupled subsequences into a latent frequency space and reconstructing the trend spectrum effectively.

While MixLinear shares some similarities with MLP-Mixer, it differs significantly in its purpose and design. The two linear layers of MixLinear in the time domain transformation serve to minimize the model scale:

• The intra-segment linear layer captures short-term patterns within segments.
• The inter-segment linear layer captures long-term patterns across segments.

Together, these layers enable MixLinear to balance local (short-term) and global (long-term) feature extraction, providing a comprehensive representation of the temporal structure.

Thank you to the authors for their efforts in addressing the issues I raised. The experimental concerns I mentioned have been resolved. Overall, as I previously noted, the proposed model is largely based on existing lightweight models, merely further segmenting the trend components and performing intra-segment and inter-segment mixing. However, the authors did not attempt to use theoretical explanations to demonstrate the effectiveness of the method, and there is a lack of intuitive visualization of the roles of the various components (trend extraction, time-domain, and frequency-domain processing). This makes the integration of these components somewhat like a mechanical patchwork. Furthermore, although the MixLinear method effectively reduces the parameter count, its unstable results (the model's performance is not always guaranteed to be the best or even the second best) make its advantages over SparseTSF, FITS, and similar methods rather limited. I am inclined to maintain the original score.

merely further segmenting the trend components and performing intra-segment and inter-segment mixing. This makes the integration of these components somewhat like a mechanical patchwork.

The key idea behind SparseTSF is its Cross-Period Sparse Forecasting technique, which focuses on downsampling original sequences to capture cross-period trend predictions. It does not involve segmenting.

In contrast, the core concept of MixLinear in the time domain lies in segmenting the trends and performing both intra-segment and inter-segment transformations. This approach enables MixLinear to effectively capture both local and global patterns within the data. While MixLinear employs downsampling as a preprocessing step—leveraging SparseTSF techniques—it builds on this foundation with additional segmentation-based transformations.

In the frequency domain, FITS primarily focuses on positive frequency components and employs a low-pass filter (LPF) to eliminate high-frequency noise. However, our approach incorporates both positive and negative frequency components, which has proven effective in enhancing prediction performance.

Furthermore, we compress the frequency components into a 2D frequency latent space. As illustrated in Figure 9a (Page 26), the first dimension of the weights predominantly captures high-frequency components, while the second dimension focuses on low-frequency components. This demonstrates that our method is more adaptive than FITS in capturing complex patterns within the frequency domain.

The foundation of this work is rooted in trend downsampling, which is the core idea behind SparseTSF (ICML 2024 Oral)—specifically, periodic downsampling and parameter reuse.

MixLinear differs significantly, not only in the patterns it learns but also in its computational complexity.

For a time series of length $L$ with a period $w$:

• The computational complexity for downsampling (as used in SparseTSF) is $O\left(\frac{L}{w} \cdot \frac{L}{w}\right) = O\left(\left(\frac{L}{w}\right)^2\right) = O(L^2)$.
• In contrast, the computational complexity for segmentation (as used in MixLinear) is $O\left(\sqrt{L} \cdot \sqrt{L} + \sqrt{L} \cdot \sqrt{L}\right) = O(L)$.

This significant reduction in complexity gives MixLinear a clear advantage over SparseTS

Comment1:
The authors demonstrate the computational complexity of MixLinear is $O(n)$, but additional operations such as Fourier Transform introduce additional complexity. A more detailed discussion on the computational complexity of MixLinear would be better.

Response:
Thank you for your comment. We have demonstrated in the paper that the computational complexity of MixLinear is $O(n)$. For model computation complexity analysis, MixLinear models long-term dependencies in the frequency domain by mapping the decoupled time series subsequences (trends) into a latent frequency space, where it reconstructs the trend spectrum. To further enhance efficiency, MixLinear applies an aggregation downsampling step using a 1D CNN with a kernel size of $w$, resulting in a computational complexity of $O(w)$. Following this, in the frequency domain transformation, FFT and iFFT operations are applied to the downsampled trend segments of length $n$, introducing an additional time complexity of $O(n \log n)$ and space complexity of $O(n)$.

In summary:

• Total time complexity of MixLinear: $O(w) + O(n \log n) + O(n)$, which simplifies to $O(n \log n)$.
• Total space complexity of MixLinear: $O(w) + O(n) + O(n)$, which simplifies to $O(n)$.

We have added this analysis to Appendix B.6 (Page 19) of the revised manuscript.

Comment2:
In Table 2, the authors report the running time of these models during training. However, it would be better to report the running time in the inference phase.

Response:
Thank you for your comment. We have measured the inference time for MixLinear, which is 0.6 ms, the smallest among all methods. We have added these measurements to Table 2 in Section 4.3 (Page 7) of the revised manuscript.

Commen3:
The experimental results in Tables 1 and 6 show that MixLinear performs competitively only on the ETTh1 and ETTh2 datasets, while its performance lags behind state-of-the-art models on other datasets.

Response:
Thank you for your comment. Our key contribution lies in significantly reducing the model size while maintaining competitive prediction performance. Many existing models are too large for deployment on end devices with limited hardware resources. In contrast, MixLinear is specifically designed to be lightweight and suitable for resource-limited devices. It provides efficient operations without compromising much on prediction accuracy.

To address your concern, we have added two more datasets (Exchange and Solar) and two additional baselines (ITransformer and SCINet) to our experiments. MixLinear continues to deliver top 2 performance on 4 out of the 6 datasets. We have updated Table 1 in Section 4.2 (Page 6) of the revised manuscript.

Comment4:
Even though MixLinear has a small parameter count, its computational complexity does not seem low. The authors need to discuss the computational complexity in more depth.

Response:
Thank you for your feedback. MixLinear models long-term dependencies by mapping decoupled time series subsequences (trends) into a latent frequency space, reconstructing the trend spectrum for effective representation of periodic variations. To improve efficiency:

• It uses a 1D CNN with kernel size $w$ for aggregation and downsampling, with complexity $O(w)$.
• FFT and iFFT operations are applied to the downsampled segments of length $n$, with time complexity $O(n \log n)$ and space complexity $O(n)$.

In summary:

• Total time complexity: $O(w) + O(n \log n) + O(n)$, simplified to $O(n \log n)$.
• Total space complexity: $O(w) + 2O(n)$, simplified to $O(n)$.

We have included a detailed analysis in Appendix B.6 (Page 19) of the revised manuscript.

Comment5:
The superscripts and subscripts in Figure 2 need to be adjusted.

Response:
Thank you for your comment. We have revised Figure 2 in Section 3.1 (Page 4) of the revised manuscript.

Comment6:
In Figure 2, it seems that MixLinear only utilizes the trend component while discarding the periodic component. Why was the periodic component discarded? Are there experimental results showing that using only the trend component is more effective?

Response:
Thank you for your feedback. The periodic components are not discarded. We perform aggregation before downsampling. For aggregation, we apply a 1D convolution with a kernel size $w$, which aggregates all information within each period at every time step. We then downsample the aggregated series by the period $w$, resulting in the trend component $X_{\text{Trend}} \in \mathbb{R}^{n}$, where $n = \left\lceil \frac{L}{w} \right\rceil$. This method effectively decouples the periodic and trend components, providing a compact representation where each trend time point encapsulates all the information from one period in the original series.

Thanks for the author's responses to my questions. I will keep my original score and will consider revising it after discussing with other reviewers.

Comment 1

There are many more time series baselines to be compared with. The number and kinds of datasets that the experiments are based on are too limited, totally 4 and 3 of them are electricity-related datasets.

Response:
Thank you for your comment. We have applied our MixLinear and baselines on two more datasets (Exchange and Solar) and examined their performance. Below are the results. We have also added two baselines (ITransformer and SCINet). MixLinear consistently provides small MSE and outperforms these new baselines. The updated experimental results have been added to Table 1 in Section 4.2 on Page 6 of the revised manuscript.

Thanks for the additional experiment results attached by the author. Your method did not consistently outperform the baseline methods, especially the FITS method. Thus, I think the performance is not sufficiently impressive and I will maintain my score.