Learning Soft Sparse Shapes for Efficient Time-Series Classification

W1: The claim is mostly supported by the ablation study. It would be better if the authors provided examples through theoretical analysis or demonstrated it with simple toy data.

A: SoftShape include:

1. Soft Shape Sparsification

• To justify its design, we provide toy real-world data (Figure 1 in the main text) with two classes: a robotic dog walking on carpet vs. cement. Existing methods discard shapes of data in a hard way, losing valuable information, whereas SoftShape uses attention-based soft weighting to preserve key shapes.

2. Soft Shape Learning Block

• Intra-shape learning: We use the MoE router to learn local class-discriminative shape features. Theoretical support: Theorem 4.2 in [1] states that the MoE router learns cluster-center features, simplifying complex issues into classification problems. Lemma 4.1 in [2] shows that MoE routers group class-discriminative patterns while filtering irrelevant patches.

• Inter-shape learning: We use a shared expert to capture global temporal patterns via multi-scale kernels. t-SNE visualizations of toy data (Figures 5 & 8 in the main text) show improved separation of mixed-class clusters over intra-shape learning.

[1] Towards understanding the mixture-of-experts layer in deep learning. NIPS, 2022.

[2] Patch-level routing in mixture-of-experts is provably sample-efficient for convolutional neural networks. ICML, 2023.

---

W2: The ablation study is only conducted on a subset of datasets.

A: We performed ablation on 128 UCR datasets, with findings consistent with the subset of datasets. Statistical results:

---

W3 & Q1: The subsequence length $m$ is not included in the hyper-parameter analysis. While all shapelets include varying lengths, SoftShape uses only one. Can you explain the trade-off of this design?

A: Using variable-length subsequences as input data raises computational complexity. Also, due to varying sequence lengths in UCR sub-datasets, larger $m$ may suit longer sequences, while smaller $m$ may be better in other cases. Following [3,4], we determine a fixed $m$ via a validation set with fewer training epochs. Experiments with a chosen $m$ reduce computation time and ensure good results, as seen in the answer to Q5.

[3] Learning time-series shapelets. KDD, 2014.

[4] CNN kernels can be the best shapelets. ICLR, 2024.

---

Q2: Can you detail how the attention mechanism used in the paper differs from self-attention and compare their pros and cons?

A: As noted in [5], the key difference is that gated attention (used in our method) assumes instance independence, enabling key instance identification via labels, whereas self-attention captures dependencies across all instances without using labels.

However, gated attention implicitly captures shape relationships via labels. Also, we use a shared expert to learn temporal patterns among shapes.

[5] Attention-based deep multiple instance learning. ICML, 2018.

---

Q3: What is the definition of $\eta$, and what is its relationship with $Num$?

A: For a time series of length $T$, the number of subsequences of length $m$ is $J = \frac{T-m}{q}+1$, where $q$ is the sliding window step. $\eta$ represents the ratio of top subsequences based on attention scores. The number of sparsified subsequences $\text{Num} = J \times \eta+1$.

---

Q4: Can you explain the implementation of the linear shape in the ablation study?

A: Replace the 1D CNN in Eq. (2) with a linear layer to convert each subsequence of length $m$ into an embedding of dimension $d$.

---

Q5: What is the effect of the hyper-parameter $m$ on the classification performance?

A: We conducted three experiments for $m$:

1. Val-Select: Choose a fixed m for one SoftShape model using the validation set.
2. Fixed-8: Set m = 8 for one SoftShape model.
3. Multi-Seq: Use fixed lengths (8, 16, 32) in parallel three SoftShape models, with residual fusion for classification.

We found no significant performance difference between Val-Select and Multi-Seq (p-value > 0.05). The Val-Select model also has fewer parameters and lower runtime.

W1 & Q2: The SoftShape model is reliance on fixed-length patch partitions for dividing the input time series, which may result in the loss of important discriminative information. Exploring the possibility of variable patch lengths might enhance the model’s flexibility and potentially improve its performance. It would be valuable to discuss this further.

A: Using shapes (or patches) of varying lengths as model inputs is intuitive but increases computational complexity and requires preprocessing for standardization. To address this, we follow [1,2] by selecting an optimal fixed-length shape via the validation set, balancing efficiency and performance. Also, SoftShape employs a shared expert with multi-scale convolutional kernels to capture dependencies across shapes, mitigating the limitations of fixed-length patches.

Further, we conducted further experiments on the 18 selected UCR datasets. The settings and results are as follows.

1. Val-Select: Choose a fixed m for one SoftShape model using the validation set.
2. Fixed-8: Set m = 8 for one SoftShape model.
3. Multi-Seq: Use fixed lengths (8, 16, 32) in parallel three SoftShape models, with residual fusion for classification.

We found no significant performance difference between Val-Select and Multi-Seq (p-value > 0.05). The Val-Select model also has fewer parameters and lower runtime.

[1] Learning time-series shapelets. KDD, 2014.

[2] CNN kernels can be the best shapelets. ICLR, 2024.

---

Q1: A more detailed explanation of the shared expert mechanism could strengthen the understanding of how it contributes to the overall performance of the model.

A: For intra-shape learning, MoE treats each shape within a time series as an independent sample, making it difficult to capture dependencies between shapes. Additionally, while learning intra-shape temporal patterns helps extract local discriminative features, it struggles to capture global temporal patterns crucial for classification. To address this, we introduce a shared expert that converts soft shapes within a sample into a sequence and leverages Inception’s multi-scale convolutions to learn global temporal dependencies.

---

Q3: The shared expert is currently not implemented with the same lightweight linear model as the class-specific expert. It might be worth considering whether using a lightweight linear model for the shared expert could simplify the architecture without negatively affecting performance.

A: We replaced the shared expert from Inception (hidden size 128) with MLP (hidden size 256) and Transformer (hidden size 512) and conducted experiments on 18 UCR datasets.

The Parameters count represents the inter-shape module (shared expert). The results show that Inception and MLP have similar parameter sizes, both significantly smaller than Transformer. This suggests that using Inception as the shared expert provides a lightweight architecture while achieving better classification performance than MLP and Transformer.

---

Q4: It would be useful to understand how λ affects the results and what range of values was tested during experimentation.

A: We analyzed the impact of different λ values on SoftShape's classification performance using 18 selected UCR datasets.

The results show that λ = 0.001 achieves the best performance, with minimal difference from λ = 0.01.

---

Q5: It could be valuable to also include a comparison with other shapelet-based methods mentioned in Section 2.2.

A: We selected ShapeConv [2] and ShapeFormer [3] as baselines. The comparison between ShapeConv and SoftShape on 128 UCR time series datasets is shown below:

Since Shapeformer requires a time-consuming shapelet discovery process before training, we provide a comparison on the 18 UCR datasets selected in the main text for Shapeformer.

The results demonstrate that SoftShape outperforms both ShapeConv and ShapeFormer on the UCR time series datasets.

[3] Shapeformer: Shapelet transformer for multivariate time series classification. KDD, 2024.

W1: The authors do not provide an in-depth analysis of the networks used for class-specific experts and the shared expert.

A: Please refer to the answers in Q2 and Q3.

---

W2 & Q4: The authors do not discuss the rationale for using warm-up training.

A: During the early training phase, the model struggles to distinguish soft shapes, resulting in the fusion of many discriminative ones. Hence, we enable shape sparsification after warm-up training. Experiments on 18 selected UCR datasets showed average ranks of 1.27 and 1.56 with and without warm-up training, respectively, highlighting the strategy's effectiveness.

---

W3 & Q1: The authors do not provide a detailed explanation of how the MoE combines the learned features from different experts to obtain the intra-shape representations.

A: For each expert's input, we use the top k indices from the MoE router to determine the activated experts for all input shapes. First, we store the shape indices in an array. After all experts output their intra-shape representations, we combine them in the original order using the stored indices.

---

W4: It would be better to compare the proposed method with LightTS, a lightweight time series classification framework.

A: LightTS uses distillation to reduce runtime by ensembling multiple models. It consists of two stages: teacher and student. The teacher phase trains at least 10 classification models (e.g., InceptionTime), which are then used for distillation to create the student model. Since the teacher phase is time-consuming, we performed a comparison using the selected 18 UCR datasets.

The results show that SoftShape outperforms both LightTS (Teacher) and LightTS (Student).

---

W5: It is encouraged to compare the proposed method with existing time series foundation models, such as UniTS and OFA .

A: OFA corresponds to GPT4TS in the main text. We compared UniTS and OFA on the UCR 128 time series dataset:

SoftShape significantly outperforms both UniTS and OFA.

---

Q2: . What impact would replacing the MoE expert as FCN or TSLANet have on the classification performance of the proposed SoftShape model?

A: Our experiments on the 18 selected UCR datasets show that replacing the original MoE expert network with FCN and TSLANet reduces SoftShape's performance.

---

Q3: How would substituting the shared expert other architectures, such as a Transformer, affect the performance of SoftShape?

A: We replaced the shared expert network with an MLP and Transformer, and the performance on the 18 selected UCR datasets is as follows:

The results show that Inception performs better.

---

Q5: In Figure 5, panel (a) appears to contain a significantly larger number (or density) of samples compared to panels (b), (c), and (d). Could the authors clarify why this discrepancy exists?

A: Fig. 5 (a) shows the representations of all shapes, while Fig. 5 (b), (c), and (d) display the representations after soft shape sparsification. As a result, the shape density in (b), (c), and (d) is lower than in (a).

---

Q6: Are there any specific modules of SoftShape for classification? Could you assess the performance of the proposed method on time series forecasting?

A: Shapelets, first introduced by Ye & Keogh (2009) for time series classification, enhance interpretability. Most existing shapelet-based methods focus on classification tasks. SoftShape, a shapelet-based method, aims to improve performance and interpretability in classification tasks. We assessed SoftShape's forecasting performance on the following datasets using MSE-based average rank under TimesNet and TS2Vec's setting.

The above results show that SoftShape outperforms TS2Vec and TimesNet, indicating its potential for time series forecasting tasks.

W1: MultiRocket-Hydra (MR-H) should be added to this comparison as a computationally efficient baseline.

A: MR-H is a combination of the Hydra [1] and MultiRocket [2] algorithm.

• Hydra uses randomly initialized convolutional kernels, grouped into $g$ groups per dilation with $k$ kernels per group. These kernels transform input time series and count the closest matches at each time point. The counts for each group are concatenated and used to train a linear classifier.

• MultiRocket applies random initialized convolutional kernels to the time series, performs standard scaling, and fits a classifier using the transformed data (default: RidgeClassifierCV).

Hydra and MultiRocket utilize randomly initialized convolutional kernels to extract time series features, in contrast to deep learning methods that rely on a backpropagation algorithm. As a result, MR-H exhibits a fast runtime.

The core modules of MR-H do not employ deep learning techniques and operate efficiently. Thus, the official implementation code of MR-H provided by the authors is based on a CPU version. Hence, the reported training and inference times for MR-H are measured using a CPU, which is consistent with other deep learning-based baselines.

Based on training time analysis from Figure 3 in the main text, MR-H took $411$ seconds on ChlorineConcentration and $64$ seconds on HouseTwenty, significantly faster than SoftShape (2743 and 280 seconds) and MedFormer (14377 and 1296 seconds). We will include a runtime analysis of MR-H in the revised version of Figure 3.

[1] Hydra: Competing convolutional kernels for fast and accurate time series classification. DMKD, 2023.

[2] MultiRocket: multiple pooling operators and transformations for fast and effective time series classification. DMKD, 2022.

---

W2: While training times are compared, a comparison of inference times and number of parameters is missing from the paper.

A: The sample length of a single time series may slightly influence the model parameter count for certain methods. The table below shows the average parameter count for the comparison methods across 128 UCR time series datasets, with $K$ denoting one thousand.

T-loss and SelfTime are excluded from the parameter count due to difficulties in obtaining the parameter information from their official implementation code. Furthermore, RDST and MR-H are not deep learning algorithms, and thus, their parameter counts are not reported.

Additionally, as baseline settings in Figure 3 of the main text, we report the inference times (in seconds) for SoftShape, MedFormer, TSLANet, InceptionTime, and MR-H on the ChlorineConcentration (4,307 samples, length 166) and HouseTwenty (159 samples, length 2,000) datasets.

It is important to note that MR-H is executed on a CPU, whereas other deep learning methods are run on an NVIDIA GeForce RTX 3090 GPU. Overall, the differences in inference time between the deep learning methods are negligible. However, we observed that SoftShape demonstrates a slight advantage in inference time on the HouseTwenty dataset, which has a longer sequence length.

---

W3: The paragraph listing the baselines used for comparisons (L#300 - L#316) need to be structured in a more informative way by indicated to which category each method belongs.

A: Thank you for your suggestions. We have classified the 15 baseline methods into two primary groups: Deep Learning-based and Non-Deep Learning-based methods. Among the Deep Learning-based methods, we have further subdivided them into two categories based on their network architecture: CNN-based and Transformer-based. This categorization will be updated in the revised version. The specific categories are as follows:

• Deep Learning-based methods:

  a) CNN-based: FCN, T-Loss, SelfTime, TS-TCC, TS2Vec, TimesNet, InceptionTime, ModernTCN, TSLANet.

  b) Transformer-based: TST, PatchTST, GPT4TS, Medformer.

• Non-Deep Learning-based methods: RDST, MR-H.