Repurposing Foundation Model for Generalizable Medical Time Series Classification

1. Novelty: The technique novelty has not been clarified in this paper: (1) The keyword "repurposing" is not explained in the Abstract section, and is only merely mentioned with "a specialized shell enriched with medical knowledge. " in the Introduction section. However, How specialized the shell is, which makes it integrate domain knowledge from diverse datasets and diagnosis tasks, has not been emphasized. (2) "curated cohort of MedTS data" has been mentioned in many times in both the Abstract and Introduction. It seems that the collection of the curated cohorts is more important than the design in "repurposing".

2. Experiment: More experiments are required to prove the effectiveness of the proposed method (1) The authors emphasize that "repurposing" enables minimal modification & lightweight parameter update for a specific task in the adaptation. However, no experiment results prove this point by comparing it with existing task-specific adaption methods. Meanwhile, the extra "repurposing" phase may introduce more parameter updates, even if the update of the "adapting" phase can be lightweight. (2) This paper takes the TimesFM as the backbone. However, it lacks a comparison with the TSA-TimesFM. (3) Lack of ablation study to indicate the effectiveness of the delicate design in the “repurposing”.

The primary weakness of this paper is its suboptimal performance on medical time series datasets compared to existing methods. Additionally, some modules and configurations are not clearly explained. Given that experimental performance is the main concern, I will begin by discussing limitations in the experiments and model comparisons.

1. In the two-stage solution, its re-purposing stage involves a training of large classifier with 8 million parameters. This parameter size is actually larger than that of some pre-trained models on the medical time series data, which raises questions about the benefits of this adaption approach.

   For instance, Cross Reconstruction Transformer (CRT) introduces a dropping-and-reconstruction pre-training paradigm. Its default setup is 6 encoder layers, 2 decoder layers, and an embedding size of 128, resulting in a model with 3.9 million parameters. Both this study and the CRT paper use PTB-XL biosignal data, which is the largest biosignal dataset used in this study. Consequently, a 4-million-parameter Transformer is sufficient for pretraining, while this adaption method needs to train a classifier with 8 million parameters. It is problematic if domain adaptation requires an even larger adapter, as this undermines the idea of efficient adaptation.

   [1] Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer. IEEE NNLS, 2023.

2. The classification performance of this paper actually is not impressive. In terms of PTB dataset, it is a very simple binary classification task. As it is simple, we did not really see recent works about pre-training or adaption make it as benchmark. For example, a simple convolution-based model with thousands of parameters can achieve an accuracy of 95%, while the baseline of RNN and SVM can also achieve over 90% [2]. However, in this paper, the proposed method and Transformer baselines are ranged from 73% to 86%, signficantly lower than exisitng methods.

   [2] ECG Heartbeat Classification: A Deep Transferable Representation. ICHI, 2018.

3. The experimental results on the PTB-XL dataset are also lower than those of existing methods. For instance, this paper references the Biosignal Transformer, which achieves a balanced accuracy of 84.21%, an AUPRC of 92.21%, and an AUROC of 76.59%. Additionally, the CRT model reports an accuracy of 87.81% and an AUROC of 89.22%. In contrast, FORMED achieves a balanced accuracy of 71.31%, an AUPRC of 63.67%, and an AUROC of 88.44%. Other related works on this dataset also generally achieve accuracy scores over 80%.

   [3] BIOT: Biosignal Transformer for Cross-data Learning in the Wild. NeurIPS, 2023.

4. The experiments also do not show the advantages of FORMED compared to Task-Specific Model (TSM). For example, the baseline among TSMs are 2% higher than FORMED. Morever, most of baselines are designed for forecasting, like Informer, Autoformer, Fedformer, which may not be suitable baseline. This paper may include Transformer-based baselines for classification tasks.

5. The datasets used in this study are relatively easy for classification tasks due to their shorter sampling lengths, ranging from 250 to 300. Specifically, this study uses PTB-XL with a sequence length of 250. In contrast, CRT employs a sequence length of 5000, and BIOT uses a sequence length of 2500, making those tasks more challenging.

6. I also have questions about the rationale behind using the foundation model TimesFM. This paper states, "Repurposing the foundation model involves changing the forecasting head to a classification head", and "Foundation models have showcased their capability in capturing general time series patterns, through pre-training on forecasting tasks". Here, the forecasting pre-training task implies using a sequence to predict another sequence, whereas next-token prediction refers to using a historical sequence to predict the next token.

   However, as far as I know, most foundation models rely on next-token prediction rather than forecasting tasks. It should be noted that TimesFM, with its forecasting-based pre-training, is an unusual case. In comparison, next-token prediction is generally better suited for learning generalizable knowledge, often enabling zero-shot or few-shot learning capabilities [4,5].

   As a result, I am uncertain about the rationale for re-purposing by changing the head, as the primary purpose of a foundation model is to capture inherent generalizable knowledge.

   [4] Language Models are Unsupervised Multitask Learners.

   [5] Language Models are Few-Shot Learners.

7. Although this study highlights the challenges of domain adaptation in medical time series data, such as inter-dataset heterogeneity, intra-dataset heterogeneity, and data insufficiency, no experiments demonstrate these aspects.

   Aside from the benchmarking results, there is a lack of informative ablation studies, and no experiments are provided to support various claims made in the paper.

8. Few typos, such as line 202 in page 4 (f->g)

This paper suffers from one critical weakness: as the performance of FORMED is in most cases very close or below that of a task-specific model, any motivation behind its use lands entirely on its alleged lightness. Unfortunately, the authors do not provide any quantitative results demonstrating the time and computational efficiencies of their method, especially when compared to task-specific and task-adapted models; they merely provide arbitrary parameter counts.

Moreover, the most common scenario for medical institutions is supposed to be adaptation of a repurposed model, as it requires the least amount of data and computational resources. Yet, the proof provided by the authors that adaptation works is very limited (fractions of a single dataset, only two competing models, and again no efficiency results).

Finally, the claims at the end of section 4.3 are scarcely substantiated: for example, the "domain knowledge" gained by the SDA during repurposing is never demonstrated.

In the current state of the paper, it is impossible to assess how significant the authors' contribution is.

Additionally:

• At line 284, the authors state that they are "processing each channel of input individually", which means that before the SDA block and thus within the backbone model where most of the weights are, channels are processed independently. This has been repeatedly reported to hinder performance on EEG and ECG data, and is even mentioned as a drawback of other models by the authors at lines 120-127.
• The bold highlight in Table 1 is unnecessary as it only denotes the best of two models. One highlighting method would be enough.
• The paper should be checked carefully for spelling and grammar errors (see below).

Miscellaneous non-exhaustive paper improvement remarks:

• Line 47, "using a pre-trained and fixed backbone foundation models", 'models' should be singular.
• Line 123/124, "in agree with Tan et al. (2024)" should read "as mentioned by Tan et al." or any correct equivalent wording.
• Line 127, "time series data and trained on multiple": a verb such as 'was' is missing before 'trained'.
• Line 131, "not suitable" can be replaced with "unsuitable".
• Line 142, "able to handle new domain of data": 'a' is missing between 'handle' and 'new'.
• Line 143/144, "on its dark side" should be more formal: 'However' is a valid alternative in this context for example.
• Line 147/148, "modification" should most likely be plural; "specific to certain task" is missing an 'a' between 'to' and 'certain'.
• Line 185/186, "The backbone foundation model is frozen in pre-training while trainable in repurposing and adapting": shouldn't this be the opposite?
• Line 284, a 'the' should be added between 'of' and 'input'.
• Line 295/296, There should be an 's' at the end of 'input', a 'a' before 'dynamic number of output classes', a 'a' before 'new dataset' and a 'the' before 'risk of overfitting'.
• Line 320, "all the parameters in SDA is independent on" should read "all the parameters in SDA are independent of".
• Line 326, "the weights in SDA is randomly", 'is' should be 'are'.
• Line 490, "despite the such inter-dataset", either 'the' or 'such' should be removed.
• Line 505/506, "on field of MedTS", a 'the' should be added between 'on' and 'field'.
• Line 537, "deserves", no final 's' is needed here.

1. Although five datasets are used to test generalizability, some datasets are quite similar, such as PTB and PTB-XL or datasets with Alzheimer’s data differing only by channel configurations. Testing on a wider range of datasets with different tasks could better showcase FORMED’s adaptability and robustness.

2. Most baseline models are Transformer-based; including comparisons with ResNet-based models, especially those tailored for MedTS tasks (like 12-lead ECG classifiers achieving strong results on PTB-XL), could provide a more comprehensive evaluation of FORMED’s performance.

3. The paper could benefit from additional experiments to deepen the analysis. For example, evaluating how FORMED performs when trained on a single dataset rather than a diverse set would provide insight into how training on multiple datasets impacts generalization. Such experiments could reveal how the model leverages knowledge across datasets and whether it shows improved performance or robustness compared to training on a single dataset alone.

4. Although Medformer is mentioned in the paper, it is not included as a baseline model. Adding Medformer to the baseline comparisons would provide a more comprehensive evaluation and better highlight FORMED's performance relative to existing MedTS models.