We answer your questions below:

• Choice of gating function: As we presented in Section 4.4 (Line 287), for a classifier trained with cross-entropy loss, the optimal $\hat{r}_i$ is a one-hot vector. The SBM is confident about its prediction on $x_i$ if $\hat{r}_i$ is sparse, and not confident if $\hat{r}_i$ is uniform. Therefore, we choose this gating function design since it is straightforward for classifiers trained with the cross-entropy loss, without the need of additional mechanisms or parameters. Within the scope of this paper, we focus on this gating function design.
• What if the expert is confidently wrong: Thank you for this question. As shown in Figure 7 in Appendix B.2., we do observe the instances where $\eta(x_i)$ is high but the prediction is incorrect. However, in all those cases, the DNN provides the same predictions. This suggests that the issue arises from a distributional shift between the training and testing samples, leading the model to overfit to the essential features present in the training data. Importantly, we did not encounter cases where the SBM was confidently wrong but the DNN was correct. This indicates that the gating mechanism effectively prioritizes reliance on the SBM for "easy" samples. Therefore, we conclude and hypothesis that the SBM would not be confidently wrong if the test samples are similarly distributed as the training samples.
• Choice of the DNN expert: we use the FCN since it has strong performance according to [1]. In the revised manuscript, we provide additional experiment results of using other DNNs, including Transformer, PatchTST, and TimesNet. We hope these results could provide a more comprehensive assessment to our method and address your concerns.
• Maximum channel: We don't think there is such a parameter of limitation on the maximum number of channels. If there is a part of the paper that caused this confusion, we would appreciate clarification so that we can address it directly.
• Memory overhead and length limitation: Our current implementation prioritizes computational efficiency using large matrix operations, which results in higher memory usage. An alternative implementation using loops would trade efficiency for memory, effectively mitigating the memory overhead. On a GPU with 32 GB memory and batch size 32, we found the time-efficient implementation work for $T < 3000$. However, with the memory-efficient implementation, running the dataset with $T>10000$ requires less than 10 GB of memory. We add more details to Appendix A.3.
• Relation between performance and $M$: The overall computational complexity of SBM is $O(MK|L|\cdot T^2)$. Therefore, the effect of $M$ is less significant than $T$.
• Parameter $\beta$: In our experiments, we used constant and cosine $\beta$ schedule with an initial value of $\beta=1$. These are heuristic designs intended to balance the training of SBM and InterpGN. However we acknowledge that the design of $\beta$ could be more flexible, using an adaptive schedule based on $\eta$ instead of fix schedules. We will consider them in future works.
• Choice of shapelet lengths: The range of shapelet lengths (5% to 80% of the series length) is intended to provide an initial universal setup for diverse datasets. Different datasets may have essential features at varying scales, so this setup serves as a general starting point. Like you mentioned, if $L$ only contains short or long length candidates, the performance will drop on datasets with essential features at different scales, which will require dataset-specific hyperparameter tuning. However, since the focus of this paper is on interpretability and InterpGN, we choose $L$ to contain various length candidates, which serves as a universal configuration for all datasets, allowing us to focus on other more important hyperparameters.
• Datasets where InterpGN perform worse: Based on results presented in Table 6, there are cases where the InterpGN perform worse than the SBM or the FCN. For cases where "SBM > InterpGN > FCN", the gap between the SBM and InterpGN can be large since the FCN cannot provide a valuable information for challenging cases.
• Choice of $\epsilon$: The default $\epsilon=1$ is chosen heuristically. A large $\epsilon$ results in tighter RBF kernels, forcing the shapelets to align more closely with actual subsequences but reducing their generalizability by overfitting to specific samples. Section 5.3 and Figure 8 provide quantitative results and a detailed analysis of the effects of $\epsilon$ on shapelet quality and model performance.

[1] Wang et al., "Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline", 2016.

Thank you very much for your valuable feedback and for taking the time to carefully read our paper. We address your questions below. If you would like further clarifications, we would be happy to continue discussions.

We address your comments on weaknesses below:

• High memory overhead: We acknowledge the concern regarding memory overhead. However, we would like to emphasize that this is a limitation of our current PyTorch implementation. With low-level optimizations, such as CUDA-based implementations, the shapelet transform can achieve efficiency comparable to conventional convolutions. We have added clarifications and discussions on this aspect in Appendix A.3.
• Limited to a fixed maximum number of channels: We are unsure about the origin of this concern, as our method does not impose an explicit limitation on the number of channels. If there is a specific part of the paper that caused this confusion, we would appreciate further clarification to address it directly.
• Linear classifier might be too simple: Interpretability favors linear explanations. For more complex relationships that cannot be captured by the SBM, the DNN expert in InterpGN is designed to handle such cases, balancing simplicity and model capacity. As discussed in Section 6, we acknowledge that more advanced interpretable models could be considered, but we consider them as future works since the a major focus of this paper is on the InterpGN instead of exploitation in shapelet-based classifiers.
• Only one type of DNN: we conduct additional experiments on more types of DNNs, including Transformer, PatchTST, and TimesNet. The results and discussions are presented in Section 5.3 of the revised manuscript. Note that within the tight discussion period, we only conduct this experiment on a subset of the UEA datasets. We will extend them to at least 26 datasets in the final version if accepted.
• Comparison with very recent methods: We have updated our experiments to include additional state-of-the-art baselines from a range of methods, many of them are published within the last 24 months. We believe these updates address your concern. If there are specific additional baselines you feel we should evaluate, we would be grateful if you could specify them.

From discussions with other reviewers, we realize there are areas where we could provide clearer answers:

• Classifier is too simple: In Appendix C of the revised manuscript, we include preliminary results on SBM using more advanced classifiers, specifically a bi-linear classifier and an attention-based classifier. These classifiers can more effectively capture relationships between shapelet concepts. Our results show notable performance improvements on some datasets when substituting the linear classifier with these alternatives.
• Evaluation on explanation quality: We provide more comprehensive qualitative results on global and local explanations in Appendix B. While we were unable to conduct a user study on interpretability at this time, we hope these additional results address some of your concerns. Additionally, we include a faithfulness estimate in Appendix B.2.1 as a quantitative metric for evaluating explanation quality.
• Small number of studies regarding interpretability: To address this concern, we provide additional qualitative results, including visualizations of global and local explanations, in Appendix B.2.2. We hope these additional results could partially address your concerns.
• Comparison with LIME: we would like to highlight that our proposed method (i.e., the SBM) fundamentally differs from LIME. The SBM is self-explainable, enabling direct interpretation without relying on a surrogate model, as LIME does. In the revised Introduction (Line 82), we add a brief discussion on existing methods for integrating DNNs with interpretable models, highlighting the differences between our approach and post-hoc explanation methods like LIME.

Thank you very much for your valuable feedback and for taking the time to carefully read our paper. We address your questions below. If you would like further clarifications, we would be happy to continue discussions.

Regarding Weakness 1

Thank you for the constructive feedback. We acknowledge the concern that in certain cases (e.g., on datasets like Handwriting), the model may collapse such that $\eta$ is consistently low.

To prevent the collapse scenario during training, we have an explicit term $\mathcal{L}_{\text{int}}$ in the overall loss function to directly optimize the SBM component (Section 4.5), separate from the cross-entropy loss of the hybrid model. This extra regularization is in addition to the gradients from the overall prediction of InterpGN and ensures that the SBM is actively encouraged to find meaningful shapelets. By directly optimizing the interpretable expert with this regularization controlled by hyper-parameter $\beta$, we aim to mitigate the risk of the model defaulting entirely to the DNN, thereby preserving interpretability where possible. In this revised manuscript, we clarify and emphasize this in Section 4.5. We also visualize how $\eta$ changes during training in Figure 12 and Figure 13 in Appendix B.4, which validate our claims.

In the explanations, we agree with the comment where we can indeed include the value of $\eta$ in the explanations. For example, the local explanation could be enhanced to state: “For time-series $x$, the confidence of the SBM is $\eta(x) = 0.8$. The sample is categorized as class X because it contains shapelets $s_1, s_2, \dots$”. This additional information helps users understand the level of confidence in the interpretable component’s prediction.

Regarding Weakness 4

We agree that more advanced classifiers, such as graph neural networks, could capture interactions between shapelets and enhance the interpretable expert, however this requires a level of complexity beyond our primary goal. One of our primary focuses in this paper is to demonstrate the effectiveness of the InterpGN framework, which aims to balance interpretability and performance when the interpretable expert may not be expressive enough. To maintain simplicity and clarity, we choose to use a linear classifier in this work, following conventional approaches in shapelet-based models. This choice allows us to clearly identify the shapelet’s contributions. In future work, we plan to explore more sophisticated models and improve the interpretable expert.

Regarding Questions on the method

• Question 2: We use Euclidean distance because it aligns with the conventional definition of shapelet-based approaches in time-series classification literature. While we acknowledge that Euclidean distance is not offset invariant, we chose it to maintain consistency with prior works. This allows for a fair comparison with existing shapelet-based methods (such as STRF, ShapeNet, and ShapeConv). Exploring other similarity measures could be an interesting direction for future work.
• Question 3: This is a great question. The datasets are designed for time-series classification tasks where labels describe the characteristics of the entire sequence rather than the temporal status at a specific point. Thus, in our current formulation, the presence of a shapelet is considered equally important regardless of its position. The existing shapelet transform does not natively capture positional information. However, one could use the timestamp of minimum distance as an additional feature alongside $p_{i, k}^{m, l}$, which would increase the expressiveness of the shapelet transform.

Regarding Questions on the experiments

• Question 4: The performance of the DNN-only model is presented in the FCN column, and the SBM-only model is presented in the SBM column in Table 1.
• Question 5: We thank the reviewer for this insightful comment. We agree that the Gini Index would be a good metric. However, we argue that the dataset-agnostics threshold values could still provide valuable insights. While the scale of weight may vary, the thresholds offer a quick heuristic to assess weight sparsity, providing a sense of how many shapelets should one interpret. In general, the Gini Index answers "how sparse are the weights" while the ratio above thresholds answers "how many". In the revised manuscript, we include both the Gini Index and the empirical thresholds for a comprehensive assessment of weight sparsity. (See Section 5.3, the Tables and Figures are moved to Appendix B.3.)
• Question 6: We appreciate that a user study could be a great evaluation tool for the interpretability aspect, and one could be considered for a simple application like the UGL dataset considered in Figure 4. However, a user study is a major effort that we cannot complete at this time. On the other hand, we also believe a stricter study on interpretability is not necessarily warranted here - our goal is to show that, given an interpretable model, we can combine it and train simultaneously with the FCN to improve the performance of the interpretable model since the focus is taken away from difficult instances that cannot be predicted by the interpretable model.
• Question 7: Thank you for raising this point. We would like to clarify that, in our model, the SBM is not a surrogate model but rather a model that contributes to both predictions and explanations. Faithfulness is typically a metric to compare the predictions of a surrogate model built for explainability to the predictions of the model that is being explained. Such metrics come up for post-hoc explainability methods such as LIME. The difference is that our model is self-explainable so we not have a a reference and surrogate model to measure the faithfulness between. As discussed in the paper, self-explainable models are meant to be directly interpretable (as per the sparsity of the classifier) and this is why surrogate models like that in LIME (a linear surrgoate) are considered interpretable versus the models they are used to explain.

Regarding other questions

• Question 8: Thank you for this question. In our formulation, we refer to $p$ as a "predicate" because it represents the likelihood that a shapelet $s$ exists within a time series $x$. By applying an RBF kernel centered at zero, $p=1$ when $s$ matches perfectly with a subsequence of $x$, and $p=0$ when they are very different, which effectively serving as a probabilistic indicator. This terminology is consistent with neuro-symbolic and interpretable models that extend the notion of predicates to include soft, probabilistic conditions, such as NSTSC (Yan et al., 2022).
• Question 9: We consider the SBM a "rule-like classifier" because its predictions are based on a linear combination of soft-logic predicates $p$. The linear classifier in SBM can be interpreted as forming logical rules of the form: “Sample $x$ belongs to class $c$ if it contains shapelet $s_1$ and does not contain shapelet $s_2$, etc.” with weights indicating the relative importance of each shapelet for the classification. This structure aligns with rule-based reasoning, where the model combines positive and negative conditions to make a prediction (Riegel et al., 2020). However, since the predicates are soft and the predictions $r$ are not bounded as logical statements, we term the predictions as "rule-like" classification.

We thank the reviewer again for their feedback and hope that we have addressed their questions. If so, we hope they will consider increasing their score.

Thank you very much for your valuable feedback and for taking the time to carefully read our paper. We address your questions below. If you would like further clarifications, we would be happy to continue discussions.

Regarding Weakness 1: interpretability of InterpGN and relationship with IME

We appreciate the reviewer’s concerns regarding the interpretability of InterpGN compared to models like IME. We acknowledge that combining outputs from both SBM and DNN can introduce a level of opacity. However, we argue that the interpretability is not entirely lost. The gating mechanism, quantified by $\eta$, reflects how much the model relies on the interpretable SBM versus the DNN. As shown in Figure 7, when $\eta$ is high, the model’s predictions align closely with the SBM, preserving interpretability.

We also appreciate the insightful comparison with IME. We fully agree that assigning a sample to a single expert during inference, as done in IME, can indeed maximize interpretability. To address this, we introduce a gating value $\underline{\eta}$ during inference of InterpGN, where only the SBM component is activated if $\eta(x_i)$ is above the gating value, i.e., $\eta(x_i) = 1$ if $\eta(x_i) > \underline{\eta}$ (see Section 4.5). With the exact training procedure as before, we experiment with $\underline{\eta} \in \{0.5, 0.6, 0.7, 0.8, 0.9, 1.0\}$. We observe that, although the average accuracy decreases if using a small $\underline{\eta}$, the difference between $\underline{\eta} = 0.5$ and $\underline{\eta} = 1$ is only 0.004 (see Section 5.3 and Figure 9). These results verify that our proposed approach can achieve a balance between interpretability and predictive accuracy, while enforcing interpretability maintains the same level performance.

Regarding model design choices

• Diversity Loss: While the ablation study in Section 5.3 shows that $\mathcal{L}_{\text{div}}$ does not have a significant impact on the main metrics, it encourages the model to learn less redundant shapelets. We think that it still introduces favorable behavior, and therefore, keep it in our model design.

• Choice of Gating Function: One of the key contributions of our approach is using the SBM as both the interpretable expert and the gating mechanism. Instead of employing an additional linear assignment model as seen in IME, we base the gating on the confidence level ($\eta$) derived directly from the SBM. This design is more interpretable because the gating value directly reflects the model’s confidence in its interpretable predictions. Unlike a linear function, which may be simple but lacks semantic meaning, our approach ensures that the decision to rely on the DNN is driven by the SBM’s confidence. Additionally, our design also saves the $O(MT)$ parameters compared to having an extra linear gating function.

Regarding open-source implementations

Thank you for pointing this out. We will release the full implementation and experiment scripts to ensure reproducibility of all the results presented in the paper.

We answer your questions below:

1. The shapelets are indeed model parameters that are learned through back-propagation. They are randomly initialized for each channel and optimized during training to capture discriminative patterns in the time-series data. We clarify this in the revised manuscript.
2. We believe the confusion stems from differentiating between building a linear classifier over distances (existing methods) vs predicates (as in SBM). We clarify this in the revised manuscript.
3. The different colors in Figure 5 correspond to different classes in the dataset. We add this clarification in the figure caption for better understanding.
4. Minor corrections: we thank the reviewer for pointing out these typos. We correct them accordingly in the revised manuscript.

We thank the reviewer again for their feedback and hope that we have addressed their questions. If so, we hope they will consider increasing their score.

LTS in the experiments

LTS is a foundational method for shapelet learning. Although it is not directly included as a baseline, we evaluate two advanced LTS variants, ShapeNet and ShapeConv, which use explicit shapelet initializations and regularization to improve upon the original LTS formulation. Their performance can be considered representative of LTS. In general, we would expect LTS to have similar performance as the SBM.

In terms of interpretability, as noted in Sections 4.3 and 5.3, LTS methods based on shapelet distances share limitations with other approaches, as they cannot automatically identify essential shapelets based on classifier weights. In Section 5.3, we compare our formulation with linear predicates, which can be viewed as a variant of LTS, providing a fair and the most appropriate comparison to our method.

Regarding metrics in Table 1

The metrics in Table 1 represent pairwise comparisons between InterpGN and a baseline method:

• "Wins/Draws" counts datasets where InterpGN outperforms or performs equally with a baseline.
• "Losses" counts datasets where InterpGN underperforms relative to the baseline.
• The "p-value" is derived from the Wilcoxon signed-rank test, measuring statistical significance. A small p-value (close to 0) indicates significant performance differences, while a large p-value suggests comparable performance.

These metrics are commonly used in classification literature, such as in SVP-T. Clarifications on these metrics are now included in Appendix A.1.

Comparison with other interpretable methods

In terms of performance, we compare with several interpretable methods, including ShapeNet, RLPAM, ShapeConv, LTS with linear predicates, and SVP-T (weakly interpretable). However, as these methods employ different types of interpretable features, direct quantitative comparisons on interpretability are challenging. Evaluating qualitative results would require user studies, which may be subjective and is a major effort that we cannot complete at this time.

Nevertheless, in Section 5.3, we compare with LTS using linear predicates, which serves as the most appropriate direct comparison to our method. Using quantitative metrics, we demonstrate that our approach outperforms this baseline.

We appreciate your comments and hope these clarifications address your concerns effectively. If there are additional points you would like us to elaborate on, we welcome further discussion.

Thank you for your comments and feedback. We address your concerns below and provide the necessary clarifications. If you would like further clarifications, we would be happy to continue discussions.

Regarding the baseline methods

Thank you for raising this point. In the revised manuscript, we have updated the list of baseline methods to align with the current state-of-the-art in deep learning methods. The performance of the new baselines is summarized in Table 1, with detailed results included in Table 6. If there are additional specific baselines you believe should be included, we would appreciate your suggestions.

Clarification on the training of SBM

We have added a clarification in Section 4.2: "The parameters to be trained include the linear classifier weights $w_{c, k}^{m, l}$ and the shapelets $s_{k}^{m ,l}$, both of which are randomly initialized."

Clarification on the design of InterpGN

To address potential confusion regarding the Mixture-of-Experts gating design in InterpGN, we provide a step-by-step clarification:

1. Given a time-series data $x_i$, the output (logit) of the SBM (interpretable expert) is $r_i$ and the output of the DNN expert is $z_i$.
2. Since the SBM is a classifier trained with a cross-entropy objective, $\hat{r}_i = `softmax`(r_i)$ is optimized toward a one-hot vector. Therefore, the sparsity of $\hat{r}_i$ can measure the confidence of SBM's predictions.
3. InterpGN is designed to use the SBM for samples with significant shapelet features while relying on the DNN to assist with challenging samples. Based on "the use of DNN should be inverse proportional to the confidence level of the interpretable model (Line 289)", we design the gating function $\eta(x_i)$ which measures the sparsity of $\hat{r}_i$.
4. In this setup, "the interpretable expert itself serves as the gating network" (Line 283), delegating work to the DNN based on its confidence. With gating value $\eta(x_i)$, the final output of the InterpGN is a mixture of $r_i$ and $z_i$, following conventional Mixture-of-Experts (MoE) methods.

We hope this clarification resolves any confusion you may have. If you could provide details and sources of the confusion, we would be happy to clarify and discuss.

Regarding the contributions of this paper

We would like to reclaim and clarify our contributions as outlined in the Introduction:

1. We introduce "a novel ating function that assigns samples to experts based on the confidence level of the interpretable expert", from which we build and train the InterpGN model. This is an essential contribution of this paper.
2. We propose "a variant of the Shapelet Transform" that improves interpretability over existing shapelet-based methods:

• Unlike traditional Shapelet Transform methods that rely on shapelet distances, our formulation uses predicates measuring the likelihood of shapelet existence. This allows for rule-like predictions where classifier weights directly indicate the importance of the existence of specific shapelets for each class.
• Our approach enables the automatic identification and ranking of essential shapelets, which is not feasible in distance-based methods.
• This design also supports global and local explainability (Section 4.3).
• As demonstrated in Section 5.3 (Table 9), the proposed RBF predicates improve both accuracy and shapelet quality compared to linear predicates.

3. We introduce quantitative metrics to measure interpretability and shapelet quality, whereas existing shapelet-based works primarily rely on selective visualization for qualitative demonstrations.

We answer your questions below:

1. Yes, shapelets are trainable model parameters. They are randomly initialized and optimized through gradient descent to capture discriminative features within the data. Specifically, $s_k^{m, l} \in \mathbb{R}^{l}$ are the trainable parameters that appear in equation (2) defining the Shapelet Transform. We add a clarification in Section 4.2.
2. Thank your for this question. In principle, our proposed models could be adapted to tasks where shapelets are useful. For example, [1] presented a method of using shapelets on anomaly detection, while the use of shapelets in forecasting seems unexplored. However, as you suggested below, if we use the models as feature selection tools, it is possible to extend them to forecasting tasks by predicting the future shapelets. We envision these extensions as valuable future works.
3. As described in Section 3, we use shapelets of varying lengths (ranging from 5% to 80% of the time-series length) to capture features at different scales. The minimum length is set to 3 timestamps because shorter sequences would either represent a single point (1 timestamp) or a straight line (2 timestamps), which can be too simple to be considered as shapes.
4. The default hyperparameters are described in Appendix A.1. Rather than fine-tuning them for each dataset, we focused on studying their impact on accuracy, shapelet quality, and interpretability in Section 5.3.
5. To clarify: the transparency level in Figure 5(c) reflects the model’s confidence. Opaque points indicate high confidence (high $\eta(x_i)$) in the SBM’s interpretable predictions, while more transparent points indicate lower confidence (low $\eta(x_i)$), leading to a greater reliance on the DNN expert for those samples. (The confusion might be that the transparency of the scatter plot is inversely correlated with transparency of the model). In the revised manuscript, to address this confusion, we update the caption of Figure 5, and replace "opaque" and "transparent" with "shaded" and "lighter".
6. As detailed in Appendix A.2, we use a balanced subset of the MIMIC-III dataset to mitigate class imbalance. While we acknowledge that overfitting to the majority class can still be an issue, our primary focus was on demonstrating interpretability. Existing approaches, such as explicit class-balancing loss, can be integrated with our models to address this issue.
7. The range of shapelet lengths (5% to 80% of the series length) is intended to provide an initial universal setup for diverse datasets. Different datasets may have essential features at varying scales, so this setup serves as a general starting point. Therefore, we focuses on other more essential hyperparameters in the ablation study.
8. Thank you for this suggestion. We agree that InterpGN could be used as a feature selection tool. Typically, for such applications, one would want to retrain the model on the selected features, which in this case are the selected shapelets. Of course, we must also acknowledge that enhanced classification performance would be subject to the types of features, and in our experiments, we observe only a small increase in performance as you noted above.

[1] Beggel et al., "Time series anomaly detection based on shapelet learning", Computational Statistics, 2019.

We thank the reviewer again for their feedback and hope that we have addressed their questions. If so, we hope they will consider increasing their score.

Thank you very much for your valuable feedback and for taking the time to carefully read our paper. We address your questions below. If you would like further clarifications, we would be happy to continue discussions.

Regarding Weakness 1

The integration of a DNN in InterpGN is motivated by the limitations of predefined interpretable features like shapelets, which may lack the expressiveness to predict marginal and complex samples (Line 82). The DNN is introduced to enhance the predictions for cases where shapelet features alone are insufficient. For example, in Figure 5 (c), shapelet features are sufficient to predict the shaded samples, while the lighter samples near the decision boundary may need extra help from the DNN.

The design of InterpGN maintains interpretability for samples with significant shape-level feature, while selectively engaging the DNN when needed. Therefore, the interpretability of InterpGN is sample-specific, preserving transparency when possible while improving performance on difficult cases. Note that for challenging cases with low $\eta$, one would not worry that the InterpGN defaults to the DNN since the interpretable features from the SBM may not be useful in the first place (i.e., we would not trust predictions from the SBM with low condidence).

Finally, we would like to argue that the combination of an interpretable model and a DNN is not a weakness but a strength to compensate for the limitations of interpretable models. The mixture-of-expert design in InterpGN enhances the predictive performance while preserving interpretability when possible.

Regarding Weakness 2

Figure 3 is intended to provide a qualitative comparison, which aligns with standard practices in shapelet-based literature. Beyond this, we have also introduced a quantitative metric to assess shapelet quality (Equation 9) and validated the advantages of using RBF-based predicates with quantitative results in Table 9.

Regarding the inclusion of shapelets for every variable, it is important to note that these shapelets are model parameters learned from scratch. Initially, shapelets are defined for all variables as an initialization strategy, ensuring broad coverage. However, through training, the model uses L1 regularization to enforce sparsity, allowing the linear classifier to automatically select only the most informative features. As shown in Section 5.3, the final learned model is often sparse.

Regarding Weakness 3

We appreciate the reviewer’s observation regarding the interpretation of Figure 6. We acknowledge that the explanations provided may appear subjective, particularly given the lack of domain-specific expertise. The purpose of Figure 6 is to showcase the interpretability of our model using shapelet-based explanations. We agree that "steady" is the wrong wording for the "OS" variable but rather that the patient didn't survive; OS has longer trends (down and up) while the patient that did survive has short fluctuations around the same mean. In practice, these interpretations would be further refined and validated by domain experts to ensure alignment with real-world understanding.

Regarding Weakness 4

Thank you for raising this point. The value of global explanations lies in providing a sample-independent understanding of the model’s learned decision rules. Unlike local explanations, which only explains the reason behind individual predictions, global explanations reveals the overall patterns and features that the model is looking for across the dataset. In practice, a global explanation can highlight generalizable insights about the dataset. Specifically, if we consider Figure 4, as suggested in the caption, the classifier can be interpreted as a "CW Circle" gesture should contain the blue shapelet but not the red shapelet in channel 1. This is an insight that applies to all instance of class "CW Circle", but of course, it can be violated in local explanations, for example, by challenging border instances.

Regarding Weakness 5

Thank you for your feedback. We acknowledge that the performance improvement in terms of average accuracy is modest, but we argue this should not be surprising. Since SBM is a linear model over shapelets (which are learned), SBM has less freedom than FCN so we are also not surprised that it underperforms FCN. Furthermore, it only underperforms by 2.2% which we find more surprising and, moreover, encouraging. The hope in adding interpretability with SBM is to at least maintain the performance of the most specialized and uninterpretable model, which is demonstrated by Table 1. A slight improvement in performance is likely due to overfitting of FCN on instances where the simpler SBM is sufficient.

We thank you again for taking our responses into consideration. We greatly appreciate your time and efforts to review our work and provide constructive feedback.