Exploring Representations and Interventions in Time Series Foundation Models

Dear Reviewer tr3o,

Thank you for your thoughtful and detailed review of our paper. We appreciate that you found our "analysis of representation similarity, pruning effectiveness, and concept steering to be detailed and that the claims made in the paper are generally well-supported."

We have addressed your concerns and questions below, and we'd be happy to address any further questions or comments. If you feel that we have addressed your concerns, we respectfully ask that you consider increasing your score.

TSFM representational redundancy: This is a great point! We found that both supervised models trained on specific datasets, and pretrained text & vision models exhibit representational similarity. While this is common knowledge for other models, prior to our work, we were unsure if this holds for TSFMs. Just like prior work, we provide no explanation behind this phenomenon, but we agree that redundancy in the training data may cause this phenomenon. More importantly, we leverage this finding to aggressively prune TSFMs, and make their use more pragmatic in real-world tasks.

Additional experiments on real-world datasets: Please refer to section Cherry-picked example from Steering in Rebuttal to Lqam.

Questions

1. Novelty of Proposed Pruning Method We assume that you are referring to this paper in the following paper [1]. Let us know if we got the wrong paper. While our approach is inspired by Nguyen et al. (2021), it differs in key ways: (1) The authors remove individual layers, whereas we prune entire self-similar blocks at once, preserving their boundaries to maintain a notion of continuity in the model’s internal representations. (2) We are also one of the first to apply this type simple block-level pruning to large pretrained Time Series Foundation Models. (3) Finally, we validate our approach on real-world tasks beyond classification, such as imputation and forecasting-demonstrating up to 52% inference speedup with minimal performance degradation, even in zero-shot settings.

[1] Nguyen, T., Raghu, M., and Kornblith, S. Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. In International Conference on Learning Representations, 2021.

2. Inference Time of Pruning: You're absolutely right that the inference time improvements reported in Table 7 were marginal—this was due to a bug in our original implementation. Initially, pruning was done by zeroing out weights, which left computation intact and yielded minimal speedups, without utilizing CUDA kernels for sparse computations. After correcting this and implementing a computational graph-level pruning mechanism that fully skips computation of pruned blocks, we observed substantial speedups, with inference time reduced by up to 52% (e.g., from 23.17 ms to 11.16 ms for MOMENT, and from 31.68 ms to 15.33 ms for Chronos). Compared to prior pruning techniques that focus on sparsifying weights or pruning individual layers, our method is simple, structurally aligned with learned representations. These updated results are included in the paper.

3. Motivation of Steering: The motivation behind concept steering is to enable controlled, post-training updates to model embeddings, allowing models to incorporate new concepts or events into predictions without requiring additional training or fine-tuning. This approach offers several practical benefits: (1) It enables users to imbue models with new or missing contextual factors after training, which is particularly valuable when models weren't originally trained on certain scenarios. For example, we can steer vital signs in healthcare based on new treatments or adjust financial forecasts based on emerging events such as positive earnings surprises. Adding these contextual factors, even as simple trends, has significant implications for improving model predictions in zero-shot and out-of-distribution scenarios. (2) Steering also supports synthetic data generation of realistic time series variations. In our experiments with the ECG Arrhythmia classification dataset (ECG5000), we demonstrate that time series classified as normal heartbeats can be steered to produce time series classified as abnormal heartbeats. Such data generation capabilities can be used to augment data for model training or generate new samples for us to better understand the decision boundaries in model predictions.

4. Risk of cherry-picking: Please refer to section Cherry-picked example from Steering in Rebuttal to Lqam.

Dear Reviewer Lqam,

Thank you so much for your time! We are glad that you found that our paper "very nicely sets up the experimental design to support their [our] claims, is comprehensive in exploring 3 different models, and presents clear and convincing evidence regarding representational similarity". We have addressed all your comments and suggestions in the current version of the manuscript.

Below, we address your concerns and questions. If you have any further comments, we'd be happy to address them. If you feel that we have addressed your concerns, we respectfully ask that you consider increasing your score.

Cherry-picked example from Steering: We appreciate the reviewer’s suggestion to include more plots to showcase the effect of steering. To demonstrate that the steering effect is not limited to a single time series example, we generated datasets with different random seeds and assessed that the same phenomena of linear separability happens, with successful steering from one group to another that is visible in the visualizations. Also, visualizations that show influence of a specific steering intervention in the output of the model also are repeatable across different samples, datasets and setups.

We also provided additional steering results with real-world data to showcase its effectiveness for other datasets in Appendix F.1. Using our proposed steering approach in MOMENT with time series from a real-world ECG Arrhythmia classification dataset (ECG5000), we demonstrate that time series classified as normal heartbeats can be steered to time series classified as abnormal heartbeats. This result confirms that concept steering can effectively guide time series into different clinically relevant category classifications, which could help enhance our understanding of time series pattern variations that guide model predictions for medical condition classification. Additionally, such steering results highlight the potential utility of the method for synthetic data generation. To provide further evidence of the effects of steering across datasets in different domains, we are currently running experiments on steering concepts in popular forecasting (ETT-h1), classification (MIT-BIH ECG dataset), and anomaly detection (UCR Anomaly Archive) experiments.

Unclear Figures 7 and 8: We agree that figures 7 & 8 can be made more clear. We have made the following edits to improve clarity.

Figure 7- The purpose of these figures is to show where in the model linear separability peaks (as measured by normalized Fisher’s LDA objective range (0,1)). X axis is the model depth - in terms of layers, Y axis is the position of the patch of time series from which we took embeddings. The more yellow the specific (layer, patch) combination the more drastic the separation was. The red line also showcases the linear separability - the idea here was that we averaged embeddings across the patch dimension, so that we have a single time series embedding at the specific layer. Here, the X axis is also model depth, while Y axis is the degree of linear separability. In addition to updating the figure caption with this information, we have clarified that the heatmap and red line use distinct y-axes, which was not immediately clear in the original figure. To improve readability, we have updated the plot by adding labeled axes: the x-axis is now labeled “Model Depth,” the left y-axis (for the heatmap) is labeled “Patch Position,” and the right y-axis (for the red line) is labeled “Fisher’s LDA.”

Figure 8 - we’ve incorporated the suggestions about improving the captions. To clarify the setup-we provided constant time series as an input to the model (not visualized, information provided in the caption), and visualize model outputs with and without steering applied, which are referred to as ‘Perturbed’ and ‘Non-perturbed’ outputs in the figure legend, respectively. As expected, without steering applied, the model outputs a constant time series signal. With steering applied, we obtain a different intended concept output, reflecting a trend, seasonality, or a combination of both, depending on the Beta parameter. For each model output, we show the raw output (lighter color) and the moving average (darker color), which helps filter out noise that is an artifact of the model. We have updated the figure caption to include this information.

Questions

1. Line 226 – What is n? The symbol $n$ refers to the number of time series samples in the dataset. Each hidden representation $h_i^{(j)}$ corresponds to the output of the $j$-th layer at $i$-th token for all $n$ samples considered, with each embedding having dimensionality $D$.

2. On Blockwise Pruning and Table 7: Thank you for your thoughtful observations. Please refer to section Inference Time of Pruning in our rebuttal to Reviewer tr30.

Dear Reviewer i5c5,

Thank you for reviewing our paper. We appreciate your recognition of our work's contributions regarding model redundancy patterns, block-wise pruning, and interpretable latent space concepts.

Given your "Weak accept" recommendation, we'd be grateful if you could let us know how we might improve our paper to strengthen your assessment. Are there specific aspects of our analysis, experimental design, or presentation that we could improve to better convey the significance and impact of our contributions?

Thank you again for your time and consideration.

Sincerely,

The Authors