Test Time Learning for Time Series Forecasting

We thank the reviewer for their comments and the opportunity to clarify the unique contributions and motivations behind our work.

1. Addressing Limitations in Handling Non-Stationary Data

The Test-Time Training (TTT) module is not merely a modification but a systematic approach to addressing the critical limitations of existing models in handling non-stationary data and adapting to distribution shifts. TTT achieves this while:

• Maintaining manageable computational complexity, avoiding the quadratic scaling issues of transformer-based models.
• Mitigating catastrophic forgetting through dynamic, instance-specific parameter updates during inference.

Our goal was to leverage these strengths of TTT to surpass the state-of-the-art in time series forecasting, which, at the time, was TimeMachine. Our design choices, including inspiration from TimeMachine’s architecture, were motivated by the need to address these challenges effectively.

2. Beating the State-of-the-Art on Multiple Datasets

The results presented in our work highlight the competitive performance of TTT, particularly in long-term time series forecasting:

• Performance Across Horizons: TTT consistently outperformed TimeMachine across multiple horizons and datasets in both regular sequence and prediction windows.
• Full Superiority on Long Windows: TTT demonstrated clear dominance in very long sequence and prediction windows, a setting where existing models, including TimeMachine, struggle to maintain robustness and accuracy.
• Dataset-Specific Results: TTT showed strong results across key datasets such as Electricity, Traffic, Weather, ETTh1, and ETTm1, with particularly notable superiority on much longer windows.

These findings underscore TTT’s ability to adapt dynamically to evolving temporal patterns, a critical advantage for long-term forecasting tasks.

3. Novelty in Addressing Long Sequence and Prediction Windows

To the best of our knowledge:

• There are currently no models specifically designed to handle very long sequence and prediction windows, a key focus of our work.
• The majority of the literature in time series forecasting has focused on the same benchmarks and models, with limited innovation in addressing the unique challenges posed by long-term forecasting.

Our experiments on extended sequence and prediction windows represent a novel contribution to this field. By demonstrating TTT’s effectiveness in these scenarios, we provide a pathway for further exploration and development of models that cater to long-term dependencies.

4. Open to Further Comparisons

We acknowledge the importance of benchmarking against relevant work. While we have reviewed the literature extensively and are confident that our results are unique in their scope, we are open to:

• Comparing TTT against any additional models that have performed similar experiments on long sequence and prediction windows.
• Providing further empirical comparisons to validate our claims and contextualize our contributions within the broader literature.

We hope this response clarifies the motivations and contributions of our work, as well as its novel focus on long-term forecasting. Please feel free to share additional suggestions or comparisons you believe would further enhance this study.

We thank the reviewer for their comments and the opportunity to clarify the performance of TTT compared to TimeMachine in the Long-Term Time Series Forecasting (LTSF) task.

1. Performance Across Benchmarks

Our results demonstrate that TTT outperforms TimeMachine on multiple horizons across the regular benchmarks, showcasing its dominance in long-term time series forecasting. Specifically:

• Overall Performance: TTT achieves superior results in both long sequence windows and long prediction windows, as detailed in Tables 4 and 5.
• Dataset-Specific Performance:
  • Electricity Dataset: TTT achieves full dominance, consistently outperforming TimeMachine across all tested horizons.
  • Weather Dataset: TTT delivers strong results, particularly for longer prediction windows, where its adaptability enhances forecasting accuracy.
  • Traffic Dataset: TTT outperforms TimeMachine on several key horizons, highlighting its ability to model complex patterns in highly dynamic datasets.
  • ETTh1 and ETTm1 Datasets: TTT demonstrates significant improvements, especially for long-range forecasting tasks, emphasizing its robustness across diverse data distributions.

2. Role of Customizable Hidden Layer Architectures

A critical factor contributing to TTT’s performance is its support for customizable hidden layer architectures, which allows for targeted improvements in temporal feature extraction. In our work:

• We explored several architectures, including small kernel convolutional designs (e.g., Conv Stacked 5) and ModernTCN, and found that:
  • Conv Stacked 5 provided significant improvements over the native TTT architectures (MLP and Linear), balancing computational efficiency with enhanced performance.
  • *ModernTCN, while theoretically promising, did not consistently outperform simpler convolutional designs, suggesting that architectural complexity alone does not guarantee better results.
• These findings highlight the potential of TTT when paired with optimized architectures tailored to specific tasks.

While we achieved substantial gains with the architectures we tested, we recognize that the design space for hidden layer architectures remains vast. Due to time and resource constraints, a full exploration of this space was beyond the scope of our work. However, the results presented here demonstrate the promise of TTT with flexible hidden architectures, and we believe that further exploration could yield even greater improvements.

3. Key Insights and Future Directions

The success of TTT over TimeMachine can be attributed to its

• Dynamic Adaptation: TTT’s test-time updates allow it to adjust to evolving data distributions, which is critical for non-stationary datasets.
• Architectural Flexibility: The ability to customize hidden layer designs enables TTT to capture both local and long-range dependencies effectively.
• Robustness Across Datasets: TTT’s strong results across diverse datasets, including Electricity, Weather, Traffic, ETTh1, and ETTm1, emphasize its generalizability.

We believe that further exploration of architectural designs and larger-scale experimentation would unlock additional potential for TTT, particularly for tasks requiring even longer horizons or highly complex temporal dependencies.

We hope this response clarifies TTT’s superior performance and highlights the potential for future architectural improvements. Please let us know if further elaboration or clarification is needed.

We thank the reviewer for their feedback and for highlighting the distinction that the TTT module is not a model itself. The primary aim of our paper was to utilize the Test-Time Training (TTT) module to address key challenges in Time Series Forecasting, specifically its ability to:

1. Handle non-stationary data effectively through test-time adaptation.
2. Learn long-range representations dynamically during inference, enabling robust performance under evolving data distributions. Our goal was to leverage these strengths to surpass the state-of-the-art (TimeMachine) while systematically exploring the impact of hidden layer architectures and providing detailed ablation studies.

1. Why We Focused on Hidden Layer Architectures?

Adding the TTT module to other architectures was not the primary aim of this work. Instead, we sought to address specific limitations in existing models commonly applied to Time Series Forecasting:

• Transformer-Based Models: While effective for capturing long-range dependencies, these models suffer from quadratic complexity with respect to the sequence length, making them computationally expensive for long-term time series forecasting.
• MLP- and Convolution-Based Models: These models are computationally efficient but are often limited in their ability to handle distribution shifts, which are common in non-stationary time series data.
• Mamba-Based Models: While efficient, Mamba’s static hidden layer architectures restrict its ability to learn complex feature representations, especially in tasks requiring dynamic adaptability.

By contrast, the TTT module combines the efficiency of linear RNNs with the ability to adapt dynamically at test time, overcoming many of the limitations faced by these architectures.

2. Why TTT Was Selected?

Our selection of TTT was motivated by its unique capability to:

• Adapt to Distribution Shifts:
  • The TTT module dynamically aligns the model’s feature representations with unseen distributions, a critical requirement for non-stationary data.
• Avoid Catastrophic Forgetting:
  • Through its instance-specific adaptation using self-supervised tasks, TTT ensures that parameter updates remain local and do not interfere with knowledge acquired during training.
• Support Flexible Hidden Architectures:
  • TTT allows for customizable hidden layer designs, enabling efficient exploration of architectures (e.g., Conv Stacked 5) that balance local temporal feature extraction with computational efficiency.

These capabilities are detailed further in:

• Appendix A: Theoretical justification for TTT’s resilience to distribution shifts and its avoidance of catastrophic forgetting during adaptation.
• Appendix E and Appendix F: A comparative analysis of different modules (e.g., TTT, Mamba, Transformer, ModernTCN) and their complexities, as well as a model-level comparison showcasing the limitations of alternative approaches.

3. Why We Did Not Focus on Adding TTT to Multiple Models?

The purpose of this work was not to exhaustively demonstrate TTT’s adaptability across various architectures but rather to:

• Show its ability to overcome specific limitations in existing state-of-the-art models for Time Series Forecasting.
• Explore how hidden layer architectures influence the performance of TTT in this domain.

By focusing on the ablation of hidden layer architectures, we provided meaningful insights into how TTT’s performance can be enhanced for long-term time series forecasting.

We kindly encourage the reviewer to refer to Appendix E and Appendix F for our analysis of various models and modules, as well as Appendix A for the theoretical motivations behind TTT’s efficacy. We hope this explanation clarifies our focus and the rationale behind our methodology. Please feel free to share any additional feedback or questions.

We thank the reviewer for their comments regarding the performance of TTT compared to TimeMachine in the Long-Term Time Series Forecasting (LTSF) task. We would like to detail how TTT compares to TimeMachine and why it performs better overall.

1. Dominance in Long-Term Forecasting

TTT outperformed TimeMachine across multiple horizons and datasets, as demonstrated in Tables 4 and 5 of the manuscript. Specifically:

• Overall Performance:

  • TTT showed superior results in both long sequence windows and long prediction windows, showcasing its strength in handling the challenges of long-term dependencies in time series forecasting.

• Dataset-Specific Performance:

  • Electricity Dataset: TTT achieved full dominance, consistently surpassing TimeMachine across all horizons.
  • Weather Dataset: TTT demonstrated strong results, particularly for long prediction windows, where its dynamic test-time adaptation enabled robust modeling of evolving weather patterns.
  • Traffic Dataset: TTT outperformed TimeMachine on key horizons, highlighting its ability to adapt to complex patterns in highly dynamic traffic data.
  • ETTh1 and ETTm1 Datasets: TTT achieved significant improvements, particularly on long-range forecasting horizons.

These results highlight the robustness and adaptability of TTT in diverse forecasting scenarios, making it a strong candidate for tasks requiring accurate long-term predictions.

2. Benefits of Customizable Hidden Layer Architectures

One of the key strengths of TTT lies in its customizable hidden layer architectures, which allow for significant flexibility in tailoring the module to specific tasks. While our exploration focused on convolutional architectures with small kernels (e.g., Conv Stacked 5 and ModernTCN), further gains could potentially be achieved by investigating more diverse architectural designs.

• Architectures Explored:

  • Conv Stacked 5: This architecture proved to be particularly effective, outperforming both TimeMachine and TTT’s native architectures (MLP and Linear) across multiple benchmarks. Its ability to enhance local temporal feature learning while maintaining computational efficiency made it a natural fit for TTT.
  • ModernTCN: While computationally efficient and theoretically promising, ModernTCN did not consistently outperform simpler convolutional architectures like Conv Stacked 5, suggesting that the balance between complexity and performance remains an area for further study.

• Future Potential:

  • Due to time and resource constraints, we were unable to fully explore the architectural design space. However, given the promising results from the convolutional architectures tested, further exploration could unlock additional improvements, especially in capturing complex temporal dynamics in long-term forecasting.

3. Why TTT Outperforms TimeMachine

TTT’s superiority over TimeMachine can be attributed to several key factors:

• Dynamic Test-Time Adaptation:

  • Unlike TimeMachine, TTT dynamically adjusts its parameters at test time to align with evolving data distributions. This adaptability allows it to handle non-stationary data effectively, which is critical for long-term forecasting tasks.

• Auxiliary Self-Supervised Task:

  • The self-supervised task used during inference helps TTT refine its feature representations, ensuring robustness to unseen variations in the test data.

• Architectural Flexibility:

  • TTT’s customizable hidden layer architectures, such as Conv Stacked 5, enable it to combine local temporal feature extraction with the global adaptability provided by test-time updates.

4. General Observations

The strong results achieved by TTT on long-term time series forecasting benchmarks suggest that its test-time adaptability, combined with well-chosen hidden layer architectures, provides a clear advantage over existing methods like TimeMachine. While our exploration of architectures was constrained, the performance improvements achieved thus far highlight TTT’s potential for further development and optimization.

We thank the reviewer for their feedback and hope this detailed explanation addresses their concerns. Please feel free to share any additional questions or suggestions.

We thank the reviewer for their feedback and the opportunity to clarify the distinction between Test-Time Learning (TTL) and Test-Time Adaptation (TTA), as well as highlight the effectiveness of Test-Time Training (TTT) in language modeling tasks.

Distinction Between Test-Time Learning and Test-Time Adaptation

Although Test-Time Learning (TTL) and Test-Time Adaptation (TTA) share the common goal of improving model performance during inference, they differ in their approaches and focus:

• Test-Time Learning (TTL):

  • TTL involves updating model parameters at test time using a loss function derived from the input data, typically in a self-supervised manner.
  • TTL is instance-specific, aiming to improve performance on individual test samples.
  • Example: Akyürek et al. (2024) applied TTL to abstract reasoning tasks, showing that parameter updates during inference improve reasoning capabilities.

• Test-Time Adaptation (TTA):

  • TTA aims to adapt pre-trained models to new, unseen data distributions during inference without relying on labeled target data.
  • TTA addresses broader distributional changes rather than optimizing for specific instances.
  • Example: Wang et al. (2023) provided a comprehensive survey on TTA approaches, showcasing their ability to enhance robustness under distribution shifts.

In summary, TTL emphasizes fine-grained, instance-specific learning, whereas TTA focuses on adapting to global shifts in data distributions. While distinct, both approaches leverage adjustments at inference to address challenges posed by non-stationary data.

Effectiveness of Test-Time Training (TTT) in Language Modeling

TTT has demonstrated significant potential in language modeling tasks, particularly in scenarios involving distribution shifts. Below are notable examples:

• Test-Time Training on Nearest Neighbors for Large Language Models (Hardt and Sun, 2023):

  • This study fine-tuned language models at test time using retrieved nearest neighbors to improve performance across various tasks.
  • TTT narrowed the performance gap between smaller and larger language models, highlighting its capacity to enhance generalization dynamically.

• The Surprising Effectiveness of Test-Time Training for Abstract Reasoning (Akyürek et al., 2024):

  • This work applied TTT to abstract reasoning tasks, demonstrating that updating parameters during inference based on input-derived loss functions improved reasoning capabilities in language models.
  • This showcases TTT’s utility in tasks requiring dynamic adaptation during inference.

These studies illustrate that TTT is not only effective for time series forecasting but also generalizes well to tasks like language modeling, where it improves performance by dynamically adjusting representations at test time.

References:

• [1] Akyürek, E., et al. The Surprising Effectiveness of Test-Time Training for Abstract Reasoning. arXiv preprint arXiv:2411.07279, 2024. Available at: https://arxiv.org/abs/2411.07279
• [2] Hardt, M., & Sun, Y. Test-Time Training on Nearest Neighbors for Large Language Models. arXiv preprint arXiv:2305.18466, 2023. Available at: https://arxiv.org/abs/2305.18466

We thank the reviewer for their thoughtful feedback. We acknowledge that the TTT module is indeed model-agnostic, and our primary motivation for using it in Time Series Forecasting was to leverage its effectiveness in handling non-stationary data, particularly distribution shifts, and its ability to capture long-range dependencies through dynamic weight updates. Our aim was to exceed the performance of the state-of-the-art at the time, namely TimeMachine.

Rather than substituting the module into multiple architectures for comparative evaluation, we focused on addressing the specific limitations of Transformer-based and Mamba-based architectures:

• Transformers:

  • While effective for long-range dependencies, their quadratic complexity with respect to the context window imposes computational constraints, especially for longer sequence lengths.

• Mamba Architectures:

  • While efficient, Mamba is less resilient to distribution shifts, which limits its robustness in dynamically evolving datasets.

Our choice of convolutional architectures was deliberate and aligned with the unique strengths of TTT. Since TTT excels at modeling long-range dependencies, we aimed to complement this by enhancing local temporal feature learning through small kernel convolutions. These architectures:

• Effectively capture local temporal features while minimizing computational overhead introduced by test-time updates.
• Include kernels of size 3 and 5, as well as stacked convolutional designs, which provide a balance between performance gains and efficiency.

Our experiments showed that these convolutional architectures significantly outperformed the native TTT architectures (MLP and Linear), demonstrating their ability to enhance the module’s effectiveness in Time Series Forecasting tasks without introducing significant computational overhead.

We hope this response provides clarity regarding our design choices and their alignment with the objectives of our work. Please feel free to share any additional feedback or questions.

We thank the reviewer for their insightful question regarding the theoretical basis for TTT’s ability to avoid catastrophic forgetting.

TTT mitigates catastrophic forgetting by performing online self-supervised learning during inference. The adaptation process is designed to operate independently for each test sample, ensuring that the original parameters learned during training remain largely intact. During test time, TTT optimizes a self-supervised auxiliary loss, which is independent of the main task loss and does not require labels or gradients from the main task. This independence ensures that updates to the model’s parameters are local and transient, specific only to the current test sample.

To further elaborate, consider the change in the main task loss due to test-time updates. Since the self-supervised task and the main task are designed to operate on orthogonal objectives, their gradients exhibit negligible interference. Orthogonality in this context implies that the dot product between the gradients of the main task loss and the self-supervised task loss is close to zero. This assumption holds because self-supervised tasks are typically constructed to exploit auxiliary structures or representations in the data, distinct from the objectives of the main task. As a result, the optimization of the self-supervised loss minimally affects the performance of the main task, preserving the knowledge learned during training.

We acknowledge that orthogonality is a simplifying assumption. While it generally holds due to the distinct nature of the main and self-supervised tasks, there may be cases where interference arises, particularly if the tasks are not well-designed. A more detailed mathematical framework and analysis, including scenarios where orthogonality might fail, are provided in Appendix A of the manuscript. This appendix also includes formal proofs and supporting empirical evidence for this theoretical claim.

We hope this clarifies the reviewer’s concerns and strengthens the understanding of TTT’s robustness. Please feel free to reach out if further elaboration is required.

We thank the reviewer for their suggestion to explore potential real-world applications of Test-Time Training (TTT). Below, we outline the practical relevance of TTT, its generalization across domains, and its unique strengths in time series forecasting.

1. Real-World Applications of TTT

TTT demonstrates significant potential for deployment in real-world scenarios, particularly in environments characterized by evolving data distributions or high non-stationarity. Some practical use cases include:

• Financial Prediction:

  • Financial markets are highly dynamic, with patterns frequently shifting due to policy changes, economic crises, or unforeseen events. TTT can adapt to these shifts in real-time using auxiliary tasks such as historical sequence reconstruction or anomaly detection.
  • Example: Predicting stock price movements or portfolio risks under conditions of sudden market volatility.

• Adaptive Traffic Monitoring:

  • Traffic patterns are influenced by external factors like weather, accidents, or public events. TTT can dynamically adjust model parameters to account for these factors, improving the reliability of traffic predictions.
  • Example: Real-time rerouting or adaptive traffic signal control during disruptions such as road closures or adverse weather conditions.

• Energy Demand Forecasting:

  • Accurate load forecasting is critical for energy systems, especially under varying conditions like temperature fluctuations or equipment failures. TTT can learn from auxiliary signals (e.g., temperature, grid stability) to adapt to non-stationary conditions.
  • Example: Predicting power demand during extreme weather events.

• Healthcare Time Series Analysis:

  • Patient monitoring involves highly dynamic data streams, such as vital signs, lab results, and environmental factors. TTT can adapt to individual patient changes during inference, improving early detection of health deterioration or anomalies.
  • Example: Predicting ICU readmissions or patient outcomes based on evolving health indicators.

2. Generalization Beyond Time Series Forecasting

While this work focuses on time series forecasting, TTT has shown promise across various sequence modeling domains, as demonstrated in prior works (e.g., Wang et al., 2023; Sun et al., 2024). Below are notable examples:

• Language Modeling:

  • In tasks like text completion or machine translation, TTT adjusts dynamically to unseen linguistic contexts during inference. Auxiliary tasks, such as masked token prediction, have been shown to improve performance under distributional shifts.

• Video Prediction:

  • TTT has been successfully applied to tasks like sequential event prediction, such as video prediction where it significantly outperforms the fixed-model baseline for four tasks, on three real-world datasets.

• Limitations of TTT Generalization:

  • While TTT is highly effective in dynamic environments, its reliance on auxiliary tasks requires careful design to align with the primary task’s requirements. In static or stationary data scenarios, TTT may introduce unnecessary computational overhead without providing significant benefits.

3. Why TTT is Best Suited for Time Series Forecasting

TTT’s unique strengths make it particularly well-suited for time series forecasting tasks:

• Handling Non-Stationary Data:

  • Time series data in domains like energy, healthcare, and traffic frequently exhibit shifting patterns due to external influences or seasonal trends. TTT dynamically adapts to these changes, ensuring robust performance.

• Capturing Long-Range Dependencies:

  • By fine-tuning hidden representations during inference, TTT enhances the model’s ability to capture both short-term and long-term patterns in sequential data.

• Robustness to Distribution Shifts:

  • Time series datasets often experience distributional changes, such as anomalies or evolving seasonal effects. TTT’s self-supervised task allows it to remain robust to such shifts without relying on labeled data.

References:

• [1] Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., & Hardt, M. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. arXiv preprint arXiv:1909.13231, 2020. Available at: https://arxiv.org/abs/1909.13231
• [2] Sun, Y., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G., Dubois, Y., Chen, X., Wang, X., Koyejo, S., Hashimoto, T., & Guestrin, C. Learning to (Learn at Test Time): RNNs with Expressive Hidden States. arXiv preprint arXiv:2407.04620, 2024. Available at: https://arxiv.org/abs/2407.04620

1. Expanding on the Choice of Hierarchical Two-Level Context Modeling

The hierarchical design of Test-Time Training (TTT) is well-suited for tasks like time series forecasting due to its ability to adapt across both high- and low-resolution contexts. Here’s why this structure benefits time-series forecasting:

• Hierarchical Representation of Temporal Dependencies:

  • Multiscale patterns in time series data, such as daily, weekly, or seasonal trends, require capturing both fine-grained and coarse-grained temporal dependencies.
  • Architectures like Conv Stacked 5 and ModernTCN implicitly model hierarchical temporal features through stacked convolutional layers and depth-wise separable convolutions, respectively. These architectures balance local temporal feature extraction with the global adaptability provided by TTT.

• Adaptation to Non-Stationary Patterns:

  • The hierarchical design ensures that the model can adapt to distribution shifts occurring at different temporal resolutions (e.g., sudden anomalies in fine-grained data vs. gradual trends in coarse-grained data).

• Supporting Literature:

  • Hierarchical approaches are widely used in time series analysis for capturing multiscale dependencies. For example, Temporal Convolutional Networks (TCN) are known to model long-range dependencies efficiently, and the hierarchical nature of TCN complements TTT’s ability to adjust to evolving data distributions dynamically.

• Proposed Benchmarks:

  • To validate TTT’s ability to adapt to multiscale patterns, we propose additional evaluations on noise-robust datasets (e.g., adding noise to ETTh1 and ETTm1) and temporal regularization tasks. Benchmarks like DeepAR or Prophet can also serve as strong baselines for comparison.

2. Comparisons with Models Handling Noise or Temporal Regularization

To position TTT relative to the state-of-the-art, we can compare its performance with models specifically designed for noise robustness or temporal regularization:

• Comparison with DeepAR:

  • DeepAR is a probabilistic forecasting model that handles uncertainty in time series data using autoregressive distributions. While it excels in forecasting under stochastic conditions, TTT’s test-time adaptation offers advantages in handling sudden, unseen distributional shifts.

• Comparison with TCN (Temporal Convolutional Network):

  • TCNs are known for their ability to capture long-range dependencies. However, they lack the adaptability of TTT, which dynamically aligns feature representations during test time. Adding noise to datasets like ETTh1 or ETTm1 could highlight TTT’s advantage over static methods like TCN.

• Theoretical Comparison:

  • TTT stands out by introducing dynamic parameter adaptation during inference. This unique feature complements models like DeepAR and TCN, which are limited to static or probabilistic adjustments. Static models like DeepAR and TCN rely on fixed parameters and are effective under stationary conditions but struggle with non-stationary data or noise. We provided more mathematical theory in Appendix A.

3. Resource Utilization: Memory, Training Time, and Inference Latency

The computational trade-offs introduced by TTT are a critical consideration, particularly in resource-constrained environments. We assess TTT’s resource utilization as follows:

• Memory Consumption:

  • TTT requires additional memory for storing gradients and activations during test-time optimization. On average, this increases memory usage by $O(T⋅d)$, proportional to the sequence length and hidden dimensionality.

• Training Time:

  • Since TTT does not modify its training procedure, the training time remains comparable to other models with similar backbones (e.g., Mamba, ModernTCN). However, inference with TTT introduces additional updates.

• Inference Latency:

  • TTT’s test-time updates increase inference latency due to gradient computations, with a total complexity of $O(U⋅T⋅d^2)$ per sample, where U is the number of updates. While this overhead is manageable in real-time systems with small batch sizes, it can be significant for high-frequency applications.

• Balancing Adaptability and Efficiency:

  • To address these trade-offs, we propose:
    • Reducing the number of test-time updates $(U)$.
    • Exploring parameter-efficient adaptations, such as low-rank updates or frozen layers.
    • Using lightweight architectures (e.g., simple convolution layers with small filters) to reduce per-sample inference costs.

We thank the reviewer for their valuable suggestion regarding a failure case analysis for TTT in Time Series Forecasting. While we did not conduct a formal failure case analysis for our specific application, we draw on insights from the original authors of TTT, who extensively evaluated its resilience under extreme distribution shifts in other domains.

In Ref [1], TTT was tested on CIFAR-10-C, a corrupted version of CIFAR-10 that includes 15 types of distortions such as Gaussian noise, motion blur, fog, and pixelation applied at five severity levels. These corruptions simulate significant distribution shifts from the original dataset. The results demonstrated that TTT significantly improved classification accuracy, achieving 74.1% accuracy compared to 67.1% accuracy for models that did not adapt during test time.

Notably:

• Gaussian Noise: Under severe shifts like Gaussian Noise, TTT effectively adapted to noisy inputs, outperforming baseline models that lacked test-time updates.
• Motion Blur and Pixelation: For distortions like motion blur and pixelation, TTT successfully reoriented the model’s feature space to handle spatial distortions.
• Comparison to Other Methods: Compared to methods such as domain adaptation and augmentation-based approaches, TTT demonstrated superior performance under extreme distribution shifts, highlighting its robustness and adaptability.

While these results focus on image classification, they provide strong evidence of TTT’s capability to handle abrupt distributional changes, which can be analogous to sudden anomalies in time series data. We acknowledge that a failure case analysis specific to Time Series Forecasting is a valuable avenue for future research and appreciate the reviewer’s suggestion.

For more detailed results, we encourage the reviewer to refer to Ref [1].

Reference:

• [1] Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., & Hardt, M. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. arXiv preprint arXiv:1909.13231, 2020. Available at: https://arxiv.org/abs/1909.13231

We thank the reviewer for raising this question regarding parameter initialization in Test-Time Training (TTT).

TTT does not require specialized or customized parameter initialization methods. For backbone architectures, TTT modules utilize standard initialization techniques, such as Xavier or He initialization, to ensure stable learning dynamics. Since TTT’s test-time updates are based on the weights learned during training, the model is agnostic to specific initialization strategies.

While TTT does not mandate particular initialization methods, it can benefit from pretrained weights. By using a pretrained backbone, the model can leverage representations already optimized for a related domain, allowing the test-time updates to refine these representations further. For example, substituting a pretrained backbone with a TTT module can enhance adaptability during inference without requiring substantial retraining.

Empirical results from prior studies (e.g., Sun et al., 2020; Sun et al., 2024) support this observation. While pretrained weights can enhance performance, they are not strictly necessary. TTT’s adaptability and effectiveness primarily stem from its self-supervised task, which guides the model to align with the test distribution rather than relying on the initialization strategy.

We hope this response clarifies that TTT is flexible and performs well across different initialization settings, with its core strength being its adaptability at test time. Please let us know if further elaboration is needed.

References:

• Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., & Hardt, M. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. arXiv preprint arXiv:1909.13231, 2020. Available at: https://arxiv.org/abs/1909.13231
• Sun, Y., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G., Dubois, Y., Chen, X., Wang, X., Koyejo, S., Hashimoto, T., & Guestrin, C. Learning to (Learn at Test Time): RNNs with Expressive Hidden States. arXiv preprint arXiv:2407.04620, 2024. Available at: https://arxiv.org/abs/2407.04620

We appreciate the reviewer’s acknowledgment of our exploration of multiple convolutional architectures. While we have not exhaustively explored all possible designs, our work focuses on the following six architectures: Conv 3, Conv 5, Conv Stacked 3, Conv Stacked 5, and ModernTCN. This subset was selected based on their potential to enhance temporal feature learning while remaining computationally efficient for test-time updates.

Due to the computational overhead introduced by TTT's updates during inference, we were constrained in the number of architectures we could feasibly implement and benchmark across all datasets. Nevertheless, our results demonstrate that convolutional architectures like Conv Stacked 5 significantly improve TTT’s performance over its native architectures (Linear and MLP). Additionally, Conv Stacked 5 outperforms TimeMachine on multiple horizons, showcasing its effectiveness for long-range time series forecasting.

We acknowledge that further exploration of hidden layer architectures is a valuable direction for future work, and we thank the reviewer for highlighting this opportunity.

Furthermore, we wish to emphasize that TTT generalizes well beyond time series forecasting. From Ref [2], TTT has been successfully applied to Language Modeling, where it demonstrated competitive results compared to Mamba and Transformer-based models. In Ref [3], TTT was used for Object Recognition, adapting to unseen event distributions dynamically. Finally, in Ref [4], TTT was extended to Video Prediction, enabling the model to adjust to environmental shifts such as changes in lighting or weather. These works collectively illustrate the generalization of TTT to other sequence modeling tasks and its effectiveness across diverse domains, including Vision Prediction, Language Modeling, and Object Recognition apart from Time Series Forecasting.

References:

• [2] Sun, Y., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G., Dubois, Y., Chen, X., Wang, X., Koyejo, S., Hashimoto, T., & Guestrin, C. Learning to (Learn at Test Time): RNNs with Expressive Hidden States. arXiv preprint arXiv:2407.04620, 2024. Available at: https://arxiv.org/abs/2407.04620
• [3] Sun, Y., et al. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. arXiv preprint arXiv:1909.13231, 2020. Available at: https://arxiv.org/abs/1909.13231
• [4] Wang, R., et al. Test-Time Training for Predicting Future Frames in Video Data. arXiv preprint https://arxiv.org/abs/2307.05014, 2023.

We thank the reviewer for their insightful question regarding the impact of test-time gradients on computational resources and our choice of hidden layer architectures.

Test-time gradients increase memory usage proportional to the sequence length $(T)$ and the dimensionality of the hidden representations ($d$), specifically scaling as $O(T⋅d)$. Furthermore, they introduce additional runtime latency proportional to $O(U⋅T⋅d^2)$, where U is the number of test-time optimization iterations. Given this computational overhead, we prioritized the use of custom hidden layer architectures that could improve model performance while minimizing additional resource demands.

To achieve this, we opted for small kernel convolutions, which effectively capture local temporal dependencies without incurring excessive computational costs. Our experiments showed that these representations significantly improved performance while maintaining computational efficiency. For instance:

• Simpler convolutional architectures, such as stacked convolutions with kernel sizes of 5 followed by 3, demonstrated strong performance improvements over the baseline architectures.
• More complex architectures, such as the ModernTCN block, while computationally feasible, did not yield significant improvements relative to these simpler convolutional designs and considerably slowed down training and inference times.

Due to time and resource constraints, we were limited in the number of architectures we could implement and benchmark across all datasets. However, the results indicate that convolutional architectures with small kernels strike a favorable balance between performance and computational efficiency. Further exploration of more advanced architectures remains a promising avenue for future work.

Additionally, we have included a detailed comparison of the computational complexities of TTT, Mamba, Transformer, and ModernTCN modules in Appendix F. A comprehensive evaluation of various models incorporating these modules is also provided in Appendix G of the manuscript.

We hope this explanation clarifies our approach and the trade-offs involved in our architectural choices. Please feel free to reach out with any further questions or suggestions.

We thank the reviewer for raising the important question about TTT’s ability to handle extreme distribution shifts effectively.

TTT leverages self-supervised tasks that are invariant to distribution shifts to guide the model’s adaptation during inference. By minimizing the auxiliary self-supervised loss, TTT dynamically reorients the model’s parameters in the feature space to align with the test-time distribution, all without requiring explicit labels. The computational overhead introduced by TTT during test time is proportional to the sequence length and the square of the hidden representation's dimensionality. Further mathematical details on this adaptation process can be found in Appendix A.

The resilience of TTT to distribution shifts has been empirically validated in Ref [1]. The authors tested TTT on CIFAR-10-C, a corrupted version of CIFAR-10 with distortions such as Gaussian noise, motion blur, fog, and pixelation applied at five severity levels, simulating significant shifts from the original data distribution. The results demonstrated that TTT significantly improved classification accuracy, achieving 74.1% accuracy compared to 67.1% accuracy for models that did not adapt during test time.

TTT’s effectiveness was particularly notable under severe corruptions, such as:

• Gaussian Noise: Representing a severe distribution shift, where TTT dynamically adapted to noisy inputs.
• Motion Blur and Pixelation: Spatial distortions where TTT outperformed baseline models by aligning the feature representations to the shifted test distributions.

In addition, TTT was compared to other approaches, including domain adaptation and augmentation-based methods, and showed superior performance under extreme distribution shifts. This highlights the unique adaptability of TTT in real-world scenarios where data shifts can be drastic and unpredictable.

For more elaborate results and a deeper evaluation, we encourage the reviewer to refer to Ref [1]:

• [1] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. arXiv preprint arXiv:1909.13231, 2020. Available at: https://arxiv.org/abs/1909.13231

We hope this addresses the reviewer’s query and provides further clarity on TTT’s robustness under extreme distribution shifts. Please let us know if additional details are required.

We appreciate the reviewer’s acknowledgment of our exploration of multiple convolutional architectures. While we have not exhaustively explored all possible designs, our work focuses on the following six architectures: Conv 3, Conv 5, Conv Stacked 3, Conv Stacked 5, and ModernTCN. This subset was selected based on their potential to enhance temporal feature learning while remaining computationally efficient for test-time updates.

Due to the computational overhead introduced by TTT's updates during inference, we were constrained in the number of architectures we could feasibly implement and benchmark across all datasets. Nevertheless, our results demonstrate that convolutional architectures like Conv Stacked 5 significantly improve TTT’s performance over its native architectures (Linear and MLP). Conv Stacked 5 also outperforms TimeMachine on multiple horizons, showcasing its effectiveness for long-range time series forecasting. We acknowledge that further exploration of hidden layer architectures is a valuable direction for future work, and we thank the reviewer for highlighting this opportunity.

We thank the reviewer for their detailed feedback. Below, we address the identified weaknesses and provide additional clarification to strengthen our work.

We agree that replacing the Mamba module with the Test-Time Training (TTT) module is not, in itself, an architectural innovation. However, our primary contribution lies in leveraging TTT's unique ability to adapt weights during test time, which enables dynamic handling of non-stationary data—a critical feature for time series forecasting tasks.

To enhance this capability, we explored customizable hidden layer architectures that improve both local and long-range temporal feature extraction. By introducing convolutional architectures (e.g., Conv Stacked 5 and ModernTCN), we enhanced the TTT module's performance beyond what is achievable with Mamba or Transformer-based backbones. These architectures provide improved local feature learning while preserving the dynamic adaptability of TTT.

Our results show that Conv Stacked 5 outperforms Mamba and TimeMachine on several benchmarks, indicating that this architectural choice adds value in enhancing TTT’s robustness for long-range forecasting tasks.

We appreciate the reviewer’s observation regarding TTT’s performance relative to TimeMachine, particularly on the ETTh2 and ETTm2 datasets. Our experiments reveal that while TTT does not always outperform TimeMachine on these datasets, it demonstrates clear advantages on long sequence windows and long prediction horizons across other datasets. Specifically:

• On ETTh1, TTT outperforms TimeMachine on the 96 and 192 horizons.
• On ETTh2, TTT shows better results on the 96 horizon.
• Across all datasets, TTT exhibits superior performance on long-range time series forecasting tasks, as evidenced by the aggregated results in Table 4 and Table 5.

To ensure the robustness of these findings, we repeated our experiments an additional four times and averaged the results. The averaged results confirm that TTT consistently outperforms TimeMachine on long sequence and prediction windows, underscoring its efficacy for long-range forecasting tasks.

We recognize the importance of providing a deeper theoretical understanding of TTT’s performance improvement. To address this, we have expanded the theoretical analysis in our manuscript as follows:

• Appendix B: A detailed comparison between the TTT module and the Mamba module, highlighting the computational and architectural differences.
• Appendix F: A comparison of the computational complexities of the TTT, Mamba, Transformer, and Fully Convolutional (ModernTCN) blocks.
• Appendix G: A comparison of different hidden layer architectures, including Conv Stacked 5, Conv 3, and ModernTCN.
• Appendix A: A theoretical discussion on how the TTT module avoids catastrophic forgetting.
• Appendix A: A theoretical discussion on how the TTT module adapts to distribution shifts along with discussion on empirical results from Reference [2].
• Appendix C: A theoretical discussion on how the TTT module compares with models handling noise or temporal regularization.

Additionally, the theoretical foundation for TTT is comprehensively discussed in Reference [1], authored by the original proposers of TTT. This reference provides a rigorous theoretical framework for TTT’s dynamic adaptation capabilities and its effectiveness under non-stationary conditions. We encourage the reviewer to refer to this work for a deeper exploration of TTT's theoretical underpinnings.

References:

• [1] Sun, Y., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G., Dubois, Y., Chen, X., Wang, X., Koyejo, S., Hashimoto, T., & Guestrin, C. (2024). Learning to (Learn at Test Time): RNNs with Expressive Hidden States. arXiv preprint. Retrieved from https://arxiv.org/abs/2407.04620
• [2] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. arXiv preprint arXiv:1909.13231, 2020. Available at: https://arxiv.org/abs/1909.13231