Abstracted Shapes as Tokens - A Generalizable and Interpretable Model for Time-series Classification

Thank you very much for taking the time to carefully read our paper. Below, we address your questions.

Regarding "why and how could you guarantee the produced attributes are accurate?"

It is important to note that our proposed method is a self-supervised pre-training model, and there are no labeled subsequences to fit. Therefore, "accurate attributes" is not a valid metric here. We can validate that the model produces "good" attributes since the decoded shapes are close to the time-series for samples not included in pre-training (see Figure 2 for an example).

Regarding "architecture-level bias of PatchTST model"

PatchTST breaks time-series into patches and projects the patches into embeddings. However, as it is a Transformer with full attention, each embedding contains non-local information. Therefore, the authors do not think that this can be a side effect that affects our proposed idea and method. The subsequence reconstruction loss in Equation 5 minimizes the difference between predicted shapes and real subsequences, which encourages learning real shapes.

Regarding "Equation 5 and loss function"

A latent code is decoded to a subsequence in the time domain. Equation 5 penalizes the difference between decoded shapes and real subsequences in the time domain. The authors are confused by the comment "reconstruction loss just a multiplication of the first term." Equation 4 penalizes the difference between the predicted time-series reconstructed from a set of $\tau_i$ and the input time-series, and Equation 5 penalizes the difference between a single predicted shape and a real subsequence. Additionally, they train different components in the model, where Equation 4 trains the time-series decoder $\mathcal{D}$ and Equation 5 trains the shape decoder $\mathcal{S}$; and gradients from them back-propagate to train the codebook and encoder.

Regarding "interpolating the input time-series"

We argue this comments with three points:

• Related works: It is common in time-series models to interpolate the time-series. For example, MOMENT subsamples long sequences to 512 timestamps and pads the short ones, and TimesNet unifies the lengths of input series to perform convolutions. Most of the common baseline methods (summarized by Wu et al., ICLR 2023) do not explicitly consider the length and sampling rate of data. Therefore, we do not think interpolating the data will create a fundamental defect in time-series classification tasks.
• For our method: Building generalizable models requires determining a generalizable way to model time-series data, such as predicting the missing patch in MOMENT and predicting the future in TimeGPT-1. In our case, we view the method of describing a time-series with 64 abstracted shapes as the generalizable way to model time-series data. Since the abstracted shapes and their attributes are defined in relative scales, we can interpolate the time-series into the same length to improve execution efficiency.
• Difference with "shapelets": In this paper, we term our codes and decoded sequences as "shapes" instead of "shapelets" since "shapelets" are originally defined as exact subsequences from the original time-series data, while our "shapes" only represent certain trend patterns. The abstracted shapes are agnostics to sampling rate and sample length. We don't think using them after interpolating the data could be an issue.

Regarding "redundant shapes in codebook and high-frequency codes"

Our current model is trained with a codebook size of 512. The codes in Figure 4 are learned from pre-training data. From the hindsight result, we see that many codes are decoded into similar shapes, which indicates that the codebook size can be reduced. We provide additional results on the model trained with smaller codebook sizes (see the Rebuttal attachment). Learning codes with high-frequency components is not guaranteed as classification datasets either do not contain high-frequency data or the high-frequency components cannot provide shape-level features. Additionally, modeling high-frequency components of time-series can be addressed by multiple tokens. However, we don't think this should be termed as a defect of the model as the codes are purely learned from data.

Regarding "comparison with TimesNet"

TimesNet is a dataset-specific model trained with supervised learning. Our model is pre-trained on multiple datasets and the weights are frozen when adapted to specific datasets with linear probing, where the linear classifier can have much fewer parameters than TimesNet. Providing useful representations that can efficiently adapt to downstream tasks is the purpose of using pre-trained models such as VQShape and MOMENT. Additionally, our method provides interpretable and generalizable representations, while TimesNet is a black-box model with end-to-end predictions.

Regarding "additional baselines and experiment results"

In the submission, we chose the best methods among each category of methods and compared them with our method. We additionally provide extensive baselines and comparisons in the Rebuttal attachment (refer to global response and Table 1). We focused on classification tasks in this paper since shape-level features can be more significant and informative in classification tasks than in other time-series tasks. We additionally provide preliminary results on extending our method to forecasting and imputation (refer to global response and Table 2 in the Rebuttal attachment).

Regarding "`the use of term token"

It is not clear what "very specific meaning" refers to, and whether it is specific to NLP. As a reference, in computer vision, VQGAN also uses token to term their codes. The authors of Large Vision Model [Bai et al., 2023] explicitly term the VQGAN encoder as image tokenizer. Therefore, we believe using the term token in our case is appropriate.

As we approach the end of discussion period, we want to make sure all your concerns are properly addressed. Please feel free to reach out if any additional clarifications are needed to assist you in future discussions and evaluations. Thanks again for your valuable time and feedback.

I wish to second this request by the authors. Dear reviewer mtPy, could you kindly comment to which degree the authors' response addressed your concerns and, if not, clarify your remaining concerns further? This would be particularly crucial since there seems to be some disagreement between reviewers on this particular paper.

Thanks for your response.

1. ""accurate attributes" is not a valid metric here." The paper claims that the attribute tuple is constructed as (code, offset, scale, relative starting position, relative length). Surely you won't have a ground truth for the code, but why don't you have any ground truth for the offset, scale, relative starting position, and relative length? I am quite confused about the choice to use MLPs to learn the later 4 attributes as well.

2. "each embedding contains non-local information" I am aware that each embedding contains non-local information, but that is not relevant to the correctness of learning shapelets. Transformers are short-sighted architectures that ignore local information and focus on global information that naturally breaks the shapelets.

3. "Equation 5 and loss function" -- Got it.

4. "interpolating the input time-series"

• It is not common in time-series models to interpolate the time-series across datasets. e.g. TimesNet unifies the lengths of input series in each UEA subset to perform convolutions, yet they never interpolate all UEA subsets (from 10+ length to 3000+ length) to the same 512 length.
• "improve execution efficiency" is an independent factor w.r.t. "learning good shapelets". If the goal is to improve efficiency, it should be ablated.

5. "redundant shapes in codebook and high-frequency codes". If the authors are arguing that classification tasks in time series in general do not require high-frequency information, that is not true and that shows you are considering a limited subset of tasks. If the authors are arguing that "not learning enough high-freq information is not a defect of the model", I do not agree. If the authors are arguing that "The current data does not have high-freq data, so they are not learnt", I would need to see experimental results that prove this point.

The other responses address my concern with limited experimental results to some extent. However, my major concern is not adequately addressed. Specifically, the paper claims to learn shapelets to improve interoperability, yet (1) the proposed method is built based on aggressive data manipulation during pre-processing, (2) the chosen architecture is quite specific. Based on many works in vision, it is reasonable to believe the chosen architecture might not be a good choice to learn shapelets; (3) According to the visualizations provided in paper, the learnt shapelets have a limited amount of diversity and thus I am not convinced by the interpretability arguments. Based on all above factors, I would keep my score the same given my current understanding.

Thank you very much for taking the time to carefully read our paper and writing a detailed review. We appreciate your comments and suggestion on giving more comprehensive statistical comparisons with the baseline. Below, we address your main concerns and your questions.

Regarding "standard deviation of experiment results"

We report the standard deviation of classification accuracy across 5 runs in Table 4 of Appendix B.1. The authors apologize for the incorrect reference in Line 393 that points to Table 2 (a summary of Table 4) for these results. We will fix this in the revised version.

Regarding "VQShape outperforms MOMENT without further information"

We follow widely used metrics to compare our methods with the baseline methods, including average accuracy, average rank, and number of top-1. Many recent deep learning methods are compared solely based on average accuracy (refer to Wu et al., ICLR 2023, Zhou et al., NeurIPS 2023). The authors agree that the statistical comparisons pointed out by the reviewer are reasonable metrics. We provide additional statistics for comparison in Table 1 of the Rebuttal attachment (also refer to the Global response for details). Based on the results, our method still achieves comparable performance as the SOTA methods in time-series classification and outperform other methods in some metrics, while additionally provides the benefit of producing interpretable features.

Regarding "performance of other models pre-trained on fewer datasets"

As the pre-training and implementation of MOMENT are not fully open-sourced, we cannot complete this experiment within the tight rebuttal window. We will try to include it in the revised version if accepted.

Regarding "clarification on Figure 3"

Channels are variates of multivariate time-series, and we will change the text in the figure accordingly. The top row consists of the histograms of the three variates averaged over all samples from the CW category, and the bottom row consists of the histograms of the three variates averaged over all samples from the CCW category. As a clarification, we will improve Figure 3 by replacing "Channel" with "Variate" and revising the caption by adding "Histograms are averaged over all the test samples from the CW and CCW category, respectively."

Regarding "codebook size"

We previously experimented with a heuristically chosen codebook size of 512. The number of "distinct" shapes could depend on the volume and distribution of pre-training data. The model can be pre-trained with a large codebook (such as 512) to guarantee expressiveness for fitting large datasets. However, the actual number of "distinct" shapes is a statistical pattern learned from pre-training, where such a conclusion can only be made afterward. We studied the effect of reducing the codebook size on downstream classification tasks and summarized the results in the Rebuttal attachment (see Table 2 and the Figures). Please refer to the global response for details. These results indicate that one can start with a small codebook size but may sacrifice performance if the chosen size is too small to fit the data.

Regarding "inductive bias that would be difficult to enforce in deep auto-encoder architectures"

Based on our understanding of this comment, we conducted an ablation by removing the subsequence reconstruction loss and reported it in Table 2 of the Rebuttal attachment (see the column with $\lambda_s=0$). With representations produced by the model pre-trained with $\lambda_s = 0$, the classifiers result in lower accuracy, indicating that the shape reconstructions actually provide additional useful information. We believe this serves as strong evidence of our advantage. However, as the shapes are abstracted subsequences, we observe that $\mathcal{L}_s$ cannot be effectively minimized to a very small value during pre-training (stuck around 0.3), which could reflect the reviewer's point on "difficult to enforce".

We thank the reviewer again for their feedback and hope that we have addressed their questions. If so, we hope they will consider increasing their score. Please let us know if our clarifications require further discussion.

We thank the authors for the response and appreciate the clarifications.

Comparisons to other methods.

We appreciate the other results for the other baselines. What exactly does the p-value in Tab.1 refer to? Specifically, with regard to what metric is it computed? Besides that, I understand that other papers make the same mistakes when comparing methods over multiple datasets. However, this does not change the fact that the additional metrics provided only give very limited insights into the actual performance difference between the methods on the datasets. As mentioned in the original review, I think it is completely fine not to show that VQShape outperforms the other methods, as it has clear advantages in terms of interpretability. However, the discussion of the experimental results in the paper should reflect this.

Baseline performance on subsets of the training datasets.

I agree that the rebuttal phase is not the time to perform such an experiment. This is why I asked for an intuition about the baseline performance in my original question, but I think this got lost in the answer. Can you provide such an intuition?

Inductive bias of VQShape.

I think the question got misunderstood. I was not asking for additional experiments but rather a discussion of the point in the paper. In my question, I mentioned that VQShape induces an inductive bias, i.e., it is possible to represent the time series via multiple shapelets. Your method induces this bias in a deep autoencoder, which evidently helps to train an interpretable model. However, the fact that VQShape induces this bias is not really discussed in the paper. In that sense, the additional ablation does not really answer the question/comment, but rather a (brief) discussion of this point should be added to the paper.

Remaining questions and comments.

While your response answered several of the initial questions, some points of my initial review were left out. This includes, for example, comments and questions regarding math notation, figure captions, etc. Could the authors briefly comment on these (e.g. clarify math notation questions ...)?

Thank you for the follow-up comments and clarifications. Below, we provide additional clarifications on our rebuttal.

Regarding "Comparisons to other methods"

The p-value is obtained from the Wilcoxon signed-rank test, which compares the rank of baseline methods with VQShape (the last column) across the UEA datasets. The p-value ranges from 0 to 1, with a small p-value indicating that the two methods (a baseline and VQShape) are significantly different, and a large p-value suggesting that the methods perform similarly on the datasets. We agree with the reviewer’s comment that the statistical significance is not strong enough to claim superior performance. Therefore, we will revise our claims in the paper to state, "VQShape achieves comparable performance with SOTA baselines while additionally providing the benefit of producing interpretable features."

Regarding "Baseline performance on subsets of the training datasets."

Thank you for the clarification. We will include more analysis on this question in the revised version, as well as experiment results to support them if possible. Our insights on this question can be summarized as:

• The pre-training datasets play an essential role in determining the quality of representations from pre-trained models, where such observations have been made in pre-trained NLP and Computer Vision models. Therefore, if pre-trained on the same datasets and having the same backbone (e.g., MOMENT and VQShape both take patch-based Transformer as the backbone), we expect the models to have similar performance on down-stream tasks.
• The difference in pre-training objective could be an important factor. MOMENT and TST employ the masked-autoencoding objective in pre-training (similar to BERT and Vision-Transformer), where each embedding will tend to capture local information. In VQShape, as we use low-dimensional code and introduce the subsequence reconstruction loss (Equation 5), the tokens and representations will learn more structured, concentrated, and non-local information. Additionally, the ablation experiment on setting $\lambda_s = 0$ is an evidence that demonstrates the subsequence reconstruction loss introduces beneficial information. Therefore, we think VQShape may also slightly outperform other pre-trained models in generalization if pre-trained on subsets of pre-training datasets.

Regarding "Inductive bias of VQShape."

We thank the reviewer for this insightful comment and apologize for any misunderstanding in our previous response. In the revised version, we will address this question by discussing the following points:

• The encoder of VQShape introduces an inductive bias that represents and summarizes univariate time-series data using a set of abstracted shapes along with their position, length, offset, and scale.
• The pre-training objectives guide the encoder toward learning interpretable (subsequence reconstruction in Equation 5) and disentangled (regularization in Equation (7) representations, while preserving the information necessary to describe the time series (reconstruction in Equation 4). These objectives mitigate the typical limitation of deep autoencoders, which often lack interpretability.
• By pre-training on diverse datasets with a universal codebook, VQShape further leverages the inductive bias to be discrete and dataset-agnostic.

We hope that these additional discussions will address your concerns and enhance the overall quality of our work.

Regarding "clarification on math notations"

We thank the reviewer for these comments to improve the clarity of our presentation and notations. In the revised version, we will clarify them as follow:

• Move line 132 to Section 3.1 and clarify "We set $l_{\text{min}}=1/64$ as it is the length of a patch."
• We think keeping the formulas for $t_k, l_k$ in Equation 1 of Section 3.2 is appropriate since the formulas are how the attributes are computed in the model; they are only scalar attributes in definitions in Section 3.1 which do not have a formula.
• $l_{\text{min}}, t_k, l_k$ are real numbers in relative scale. We apologize for the misalignment in notations. Line 113 will be updated as $0 \leq t \leq 1-l_{\text{min}}$ and Line 114 will be updated as $l_{\text{min}} \leq l \leq 1-t_k$. For simplicity in notations, we will clarify on Line 106 with "In this paper, $x^m_{i, t_1:t_2}$ denotes a subsequence between timestamp $\lfloor T t_1 \rfloor$ and $\lfloor T t_2 \rfloor$ where $t_1, t_2\in [0,1]$ are relative position."
• On Line 129, we will clarify that $\hat{\tau}$ represents $\tau$ before quantization.

Please let us know if our clarifications require further discussion.

Thank you very much for taking the time to carefully read our paper. We are glad you found our method provides interesting insights. Below, we address your main concerns and your questions.

Regarding "why excluding the InsectWingBeats dataset"

We discussed the reason at Line 393 in Appendix A. The InsectWingBeats dataset contains very short time-series, such as 1 or 2 timestamps, which do not contain any meaningful shape-level features. Considering the number of samples and channels of this dataset is significantly higher than other datasets (refer to Table 3 in Appendix A), we believe this dataset may pollute the pre-training of our model and therefore excluded it from the experiments.

Regarding "choice of benchmarking datasets"

We chose to perform quantitative evaluation on the UEA datasets because they are a default choice in most recent works [Zuo et al., AAAI 2023; Wu et al., ICLR 2023; Zhou et al., NeurIPS 2023] on time-series analysis and suggested to benchmark deep learning methods (Deep Time Series Models: A Comprehensive Survey and Benchmark, Zhou et al., 2024). We agree that adding UCR can make the experiments more comprehensive. However, the results for all the 128 UCR datasets are not usually available for the baseline methods (which also means including UCR results is not generally a mandatory requirement); obtaining the results will require additional time and we could not finish them within the Rebuttal period. We will include the results of our methods and baselines in the revised version.

Regarding "number of baselines to compare"

We chose TimesNet, T-Rep, and MOMENT as our baselines since they are the best-performing methods among supervised, representation learning, and pre-trained models. We agree that it is good to include more baselines, and we have included detailed results and comparisons with 14 baseline methods in Table 1 of the Rebuttal attachment (please also refer to the global response). Compared with more baselines, our method still achieves comparable performance as the SOTA methods in time-series classification and outperform other methods in some metrics, while additionally provides the advantage of producing interpretable features.

Regarding "the impact of reducing the size of codebook"

We studied the effect of reducing the codebook size on downstream classification tasks and summarized the results in the Rebuttal attachment. Please refer to the global response for details. As a summary, pre-trained with codebook size 64, the Histogram representations of VQShape achieve the best average classification accuracy of 0.715, which outperforms the SOTA baselines. This matches our hindsight discovery in Figure 5 of the paper where there are roughly 60 clusters of codes.

Regarding "how is the size of the codebook determined"

We previously experimented with a heuristically chosen codebook size of 512. The number of "distinct" shapes could depend on the volume and distribution of pre-training data. The model can be pre-trained with a large codebook (such as 512) to guarantee expressiveness for fitting large datasets. However, the actual number of "distinct" shapes is a statistical pattern learned from pre-training data, where such a conclusion can only be made afterward.

Regarding "how can you encourage the decoded shapes to be as diverse as possible?"

In this paper, we use the entropy terms in Equation 6 to encourage the usage of all codes in the codebook, which also promotes the diversity of latent codes. To encourage the decoded shapes in the time domain to be diverse, we can apply additional regularization by adding

to the overall loss function, as introduced by ADSN [Ma et al., AAAI 2020]. However, introducing an additional objective may make pre-training more challenging. We will leave it as future work as it is not the focus of the our current method.

We thank the reviewer again for their feedback and hope that we have addressed their questions. If so, we hope they will consider increasing their score. Please let us know if our clarifications require further discussion.

As we approach the end of discussion period, we want to make sure all your concerns are properly addressed. Please feel free to reach out if any additional clarifications are needed to assist you in future discussions and evaluations. Thanks again for your valuable time and feedback.

Thank you very much for taking the time to carefully read our paper, recognizing our contributions and giving valuable feedback. Below, we address your questions.

Regarding "Generalization experiment of MOMENT and TST"

The authors thank the reviewer for this suggestion. However, as the pre-training and implementation of MOMENT are not fully open-sourced, we cannot complete this experiment within the tight rebuttal window. We will try to include this experiment in the revised version if accepted.

Regarding "Ablation on subsequence reconstruction loss"

If removed, the codes will not have a connection with shapes in the time domain, and the model will be similar to a regular VQVAE, losing the ability to provide interpretable features. We conducted this ablation study and reported the new results in Table 2 of the Rebuttal attachment (see the column with $\lambda_s=0$). With representations produced by the model pre-trained with $\lambda_s = 0$, the classifiers result in lower accuracy, indicating that the shape reconstructions actually provide additional useful information. We believe this serves as evidence of our advantage.

Regarding "increase code size"

Our motivation for using low-dimensional codes is stated in Lines 135 to 138. Increasing the code dimension will enlarge the bottleneck and possibly create a defect that the codes contain hidden information beyond the decoded shapes. We conducted this experiment and reported it in Table 2 of the Rebuttal attachment (see the column with $d^{\text{code}}=32$). Increasing the code dimension does not significantly improve the performance of classifiers. We acknowledge that the scaling pattern for code dimension may require more extensive study, but we are unable to complete additional experiments within the Rebuttal period.

Regarding "$l_{\text{min}}, h_k, \tau_k$"

l_\min can be viewed as "the minimum length of a meaningful shape." We chose l_\min = 1/64 since the input series are transformed into 64 patches, indicating that a series with a length of $1/64$ is the basic unit for expressing a time-series. As mentioned in Line 148, $h_k = `Linear`(\tau_k)$, which is a linear embedding layer to convert $\tau_k$ into embedding $h_k$.

Regarding "Performance comparison on zero-shot evaluation with other approaches"

There are a limited number of pre-trained time-series models applied to classification tasks, as many of them are designed for forecasting (e.g., Google-TimesFM, IBM-TTM, etc.). Due to limitations in published results and open-sourced implementation (MOMENT released a checkpoint but not the pre-training code) we are unable to reproduce the results within the Rebuttal period. We will additionally include a comparison with GPT4TS [Zhou et al., NeurIPS 2023] in the revised version if accepted.

We thank the reviewer again for their feedback and hope that we have addressed their questions. If so, we hope they will consider increasing their score. Please let us know if our clarifications require further discussion.

As we approach the end of discussion period, we want to make sure all your concerns are properly addressed. Please feel free to reach out if any additional clarifications are needed to assist you in future discussions and evaluations. Thanks again for your valuable time and feedback.