Thank you for insightful comments and the extensive review. We appreciate your question on the justification for concept bottleneck models over post-hoc methods.

Our motivation for the concept bottleneck framework over post-hoc interpretability is two-fold. Firstly, the post-hoc interpretability methods are notorious for not being faithful to the model’s mechanisms. By enforcing interpretability already at training time, we attempt to overcome this and make the model interpretable by design.

Secondly, post-hoc interpretability turns out to be difficult when trying to localize specific features (or concepts) within the model. For example, we can assume any trained Autoformer model should have learned some concept of time, yet, the CKA scores do not show any specific head to have a high similarity to this (see Figure 10a in Appendix F). In practice, these high-level concepts seem to be distributed amongst different model components. (In some sense, our approach is thus complementary to the current popular idea of using Sparse Auto-Encoders (SAE) for interpretability: while SAE are often used to deal with individual neurons having different functions (i.e., “polysemanticity”), our approach manages to reduce the distributed nature of representations).

The highly distributed representation of concepts also makes it difficult to compare different model architectures, and to find out what they exactly picked up from the data. Additionally, locality of concepts can be beneficial if these concepts are important to understand and gain control over the model’s internal mechanisms, so that an intervention can be done (as we show in our intervention experiment).

We see the potential for actionable results that indicate how the input should change to obtain a different outcome. And we agree with you that the model’s insights would be more actionable when localized patterns are captured. However, that is outside the scope of our current paper, which is rather about understanding what a model learns from the data and how we can influence it.

Thanks for the useful comments, and for your appreciation of the novelty of our approach. We will briefly try to answer your questions:

• Comparison with other XAI techniques
  Comparing the interpretability of our framework with other models is not straightforward, because other interpretable time series models are conceptually orthogonal i.e., they often explain their predictions in terms of features of the input data, for example by perturbations [1] or attention scores [2, 3]. To the best of our knowledge, our work is the first to enforce a transformer to learn pre-determined concepts from the data, and use those in the down-stream task. Therefore, we cannot compare our interpretability of these concepts with another method. Essentially, we cannot use another method to enforce the model to learn a concept in a specific head, but our own (see also our response to the reviewer RmBn).
• Purpose intervention experiment
  The aim of the intervention is two-fold: first to show a possible real-world application of the concept bottleneck framework, and secondly to verify the localization of concepts. By showing the timestamps intervention works, we verify that the concept of timestamps is indeed located in the intended head.
• Choice of interpretable concepts
  The reason for the employed interpretable concepts (i.e., AR and the timestamps) is mainly due to the fact that they are domain-independent, and therefore should work well across all datasets, in a respective manner. One could also consider potentially more sophisticated concepts, such as holidays, or special events, however we wanted the concepts to be representative and insightful for all datasets.
• Validation of interpretability
  Our validation of the interpretable concepts is done with the intervention experiment. We do not conduct any user studies, even though we agree that these could be very valuable, because whether explanations are meaningful to the end user is an entire field of its own. We merely focus on providing a technical framework.
• Choice of Autoformer architecture
  The Autoformer architecture was chosen because it is a well-studied prominent Transformer model for time series forecasting. The framework architecture can be applied just as well to other Transformer architectures. We are in fact planning to apply it to other models in follow-up experiments, as part of our research agenda.
• Bottleneck location
  We compared two locations for the bottleneck: in the attention vs. the feed-forward component, both in the encoder. Overall, we find that the feed-forward bottleneck performs slightly better for most datasets (see Table 1). We focus on modelling the encoding of interpretable concepts, so choosing a bottleneck location in the decoder does not align with that idea.
• Sensitivity to the alpha hyper-parameter
  Results about sensitivity to the hyperparameter alpha are given in Appendix F. The training does not seem to be overly sensitive to the hyperparameter: That is, even with a relatively high importance to the CKA loss (high value for alpha), the model is able to achieve low forecasting errors.
• Computation overhead
  The extra computation from our method arises from calculating the CKA loss, which depends on the CKA score. This score (using a linear kernel) has quadratic complexity in the sequence length $n$, while the Autoformer has $n \text{ log } n$ complexity.
• Quantitative evaluation of interpretations
  Quantitatively evaluating the interpretations is tricky, because there is no ground truth. While we use the CKA scores to evaluate, we also make use of the intervention to illustrate that the timestamps concept is indeed encoded by the head with the high CKA score to time.
• Generalizability of intervention to other types of shift
  The limitation to the intervention is that one should have access to the shifted data, and know in which concepts the shift has occurred, so that the hidden representations from these concepts can be replaced. While this can be very well applied to a temporal shift, the notion of intervention is not limited to it.

[1] Enguehard, J. (2023). Learning Perturbations to Explain Time Series Predictions. ICML 2023

[2] Davies, H. J., Monsen, J., & Mandic, D. P. (2024). Interpretable Pre-Trained Transformers for Heart Time-Series Data. arXiv preprint arXiv:2407.20775.

[3] Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting 2021

Many thanks for the useful comments! We try to address the 4 potential weaknesses you mention, and hope to convince you to convince the scores somewhat.

W1: Comparative analysis of interpretability

We agree that comparing the interpretability of our framework with other approaches would be very interesting, but doing such a comparison is tricky. The main reason is that other interpretable time series models typically explain their predictions in terms of features of the input data, for example by perturbations [1] or attention scores [2, 3]. To the best of our knowledge, our work is the first to enforce a transformer to learn pre-determined concepts from the data*, and use those in the down-stream task.

In our study, we concluded that the best comparison is between a Transformer model before and after applying the “enforcing” interventions from our framework. For these results, we refer to Figure 10a in Appendix F. The visualization shows that the model components show limited similarity to AR and the different time concepts, whereas the similarity increases when applying our framework.

W2: More extensive research analysis

It is, of course, difficult to argue about how ‘derivative’ new work is. We just note that there has been much interest in both concept bottleneck models and time series interpretability (as we review in the paper), but that the combination we propose – which builds on insights from a diversity of subfields (including mechanistic interpretability and CKA) has not been proposed before. We have tried to do all the necessary analyses to support our claims!

W3: Impact AR in preventing model degradation

Many thanks for raising this interesting question: does the AR surrogate model make up for any loss in performance introduced by the concept bottleneck? We did, in fact, perform an experiment that partially answers it. We trained an Autoformer without the AR concept, but with the time concept and a free head. The performance on the electricity data for this model is (MSE: 0.206, MAE: 0.321), which is seemingly identical to the original performance of (MSE: 0.207, MAE: 0.320). This suggests that it is not the AR head that makes up for the loss in performance. When looking at the CKA plots, we find that the free head in the minimal set-up (without AR) has less similarity to the time concept than in the original set-up, indicating that it learns less similar representations than before. So, instead of adding performance to the bottleneck model, we believe these results show that the AR model just adds interpretability, which is in line with the claims of our paper.

W4: More complex datasets

We agree that the use of more datasets would be interesting, but would argue that they are not at this stage necessary to test the robustness of the framework. We did not cherry-pick our datasets; rather, we used the full suite of commonly used datasets from the recent time series literature (including those in the original [celebrated] Autoformer paper, including datasets [Traffic and Electricity], for which the Autoformer model outperforms AR). In fact, there is an ongoing discussion about the application of Transformers for time series (see our response to reviewer J9av).

[1] Enguehard, J. (2023). Learning Perturbations to Explain Time Series Predictions.ICML 2023.

[2] Temporal fusion transformers for interpretable multi-horizon time series forecasting, International Journal of Forecasting 2021

[3] Davies, H. J., Monsen, J., & Mandic, D. P. (2024). Interpretable Pre-Trained Transformers for Heart Time-Series Data. arXiv preprint arXiv:2407.20775.

*Note that there is work on concept-based anomaly detection for time series (Ferfoglia, I., Saveri, G., Nenzi, L., & Bortolussi, L. (2024). ECATS: Explainable-by-design concept-based anomaly detection for time series. ArXiv, abs/2405.10608.) However, this work represents concepts as Signal Temporal Logic formulae, such as, “the temperature should never exceed a certain threshold for more than a specified duration”. In contrast, by ‘concepts’, we mean high-level features from the time series data.

I have read the authors' response and thank them for taking the time to address my points.

A few comments:

W2:

Taken from the author's response to my point:

We have tried to do all the necessary analyses to support our claims!

Taken from my review:

Specifically, other proxy tasks for the interpretable concepts could have been explored, as well as other components (e.g. bottleneck location, similarity metric used, transformer models...).

To be clear, I have no issues with the work being potentially somewhat derivative, as I state. My issues is that among the free variables of the problem, not many are sufficiently explored. Does a single, arguably widely recognized transformer paper from 2021 consitute a sufficient exploration into the possible transformer backbones that could be used? The same goes for the other components, which my review was an invitation to explore.

W4:

Yes, I agree that there is extensive debate about the validity of transformers for time-series forecasting, and arguably part of that debate stems from the fact that perhaps the datasets we commonly evaluate such models on are too limited to draw robust conclusions. The FreDO paper that I mention, e.g. shows that since such datasets commonly have a strong frequency component, a non-parametric model is already good at forecasting on them. This does not mean that such a baseline would hold for all conceivable datasets, hence my question.

Respectfully, I do not feel the authors have answered my point when they mention that they have used the datasets commonly found in the literature. They set out to show that their approach brings value. I mention that it may be caused by dataset bias, and wonder if such performance would remain in some datasets for which AR is not such a strong candidate. Merely stating that the common datasets are indeed the ones that other papers tend to use does not, in my opinion, address this. This ties into my above point (W2) about insufficient exploration of the hyper-parameter/problem space.

W1

I agree with the reviewer's comments about the relatively novel nature of the problem, and their justification.

W3

This experiment seems to go in the right direction towards adressing this point. Could I just ask the authors to explain in more detail what they mean by this:

We trained an Autoformer without the AR concept, but with the time concept and a free head. (I want to be sure that I understand correctly the procedure here, esp. the free head).

Thanks for the useful comments! We hope that we can address the two worries you mention, and convince you to increase the scores slightly.

(1) Regarding your point about the simpler AR model often being the best: yes, we agree this is fairly interesting. In fact, there has been much discussion about the benefits of Transformers for time series forecasting, because simpler models seem to outperform them for some datasets. While many works (e.g., [1] next to milestone approaches such as Informer, FedFormer, and others) are in favour of employing Transformers for time series, others are not (e.g., [2], and the response from Huggingface [5]). Moreover, this is only a part of more general ongoing discussions regarding machine learning vs. statistical methods (see the influential Makridakis et al. [3], and often-mentioned Nixtla experiments [4]) which has already been a part of time-series forecasting literature for last couple of years, and likely will continue to be so.

In our paper, we tried to sidestep all these discussions, and rather focus on the Transformer's interpretability. Therefore, we do not make claims about advantages of Transformers in all time series data, but do argue that IF they are used, then interpretability is a major issue that needs to be addressed.

We focused our analyses on cases where the Transformer does outperform other models (e.g. the traffic and electricity dataset), and showed that our concept bottleneck framework is applicable. Therefore, we do not consider it a weakness that AR sometimes outperforms the Autoformer or not. We do test the framework for all these datasets, because they are often used in the time series literature (including the original Autoformer paper).

(2) Regarding the optimal setting for the alpha hyper-parameter: we find that the model performance is not heavily dependent on alpha. Recall that alpha indicates the weight of the CKA term in the loss function, and we find that so long as the CKA term is not equal to the loss function (i.e. alpha = 1), then there is a term which pushes the model to learn to forecast well. This is in line with the overall results, including the bottleneck (alpha > 0) does not decrease the original model performance (alpha = 0), and this holds for all datasets. Additionally, we would like to point out that almost all results for alpha < 1 in Table 5 from Appendix F are within the same range by standard deviation, so the best and second-best settings do not carry that much of weight.

[1] Niu, P., Zhou, T., Wang, X., Sun, L., & Jin, R. (2024). Attention as Robust Representation for Time Series Forecasting. ArXiv, abs/2402.05370.

[2] Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2022). Are Transformers Effective for Time Series Forecasting? AAAI Conference on Artificial Intelligence.

[3] Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilis & Semenoglou, Artemios-Anargyros & Mulder, Gary & Nikolopoulos, Konstantinos. (2022). Statistical, machine learning and deep learning forecasting methods: Comparisons and ways forward. Journal of the Operational Research Society. 1-20. 10.1080/01605682.2022.2118629.

[4] https://github.com/Nixtla/statsforecast/tree/main/experiments/m3

[5] https://huggingface.co/blog/autoformer

We have now performed the extra experiments you suggested with a synthetic dataset. To briefly recap: you proposed to train the model on a synthetic dataset, constructed with the concepts (plus noise) of our choice, to understand how the model leverages the concepts. In our earlier experiments, this was hard to achieve, because the best and second-best values of the hyperparameter alpha were not close in value, and therefore not intuitive (almost all results for $\alpha$ < 1 in Table 5 from Appendix F are within the same range by standard deviation, so the best and second-best settings do not carry that much weight. We included a new figure 10 in Appendix F that clearly illustrates this).

The results from the new experiments are presented in Appendix I. We generate a time series dataset as the sum of different sine functions, and then train an Autoformer model with a bottleneck on the attention heads of the second layer. We vary the value of hyperparameter $\alpha$, and define each concept in the bottleneck as one of the underlying functions (for which we have the ground truth by construction).

As expected, we find that the similarity between the bottleneck components and the concepts increases with increasing $\alpha$ (this is visible as the emergence of a yellow diagonal in layer 2 in Figure 23). At $\alpha=0$, there is no concept bottleneck and the similarity to the predefined concepts is minimal. At $\alpha=1.0$, the model is only optimized for similarity to the concepts, and the prediction performance is terrible. Interestingly, at $\alpha=0.8$, we hit a sweet spot where similarity to the predefined concepts is high and the prediction performance is also at its maximum.

We believe these additional experiments help in understanding how the model leverages interpretable concepts. We would like to thank you again for the suggestion, and are curious to hear whether it is indeed exactly what you had in mind.