Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classification

Thank you for your thoughtful feedback on our work！ We appreciate you carefully reading our method, equations, figures, and even the code! We are happy to receive your endorsement and feel that our paper is easy to follow! Again, thank you for your comments. Here are the responses to your questions and concerns.

Q1: Processing medical time series should provide explanability and transparency.

A1: Thanks for raising this concern. We agree explanability is crucial to medical tasks, therefore we further clarify how we can provide explanations from our trained model with an example from the TDBrain dataset. This is a dataset of EEG signals for prediction on Parkinson's disease, which is suitable for exploring if our model is utilizing the correct channels' information and the signals at the right frequency for a reasonable diagnosis. To find out important channels contributing to our model's prediction, we first mask out the rest of the brain regions and only use certain combinations of them to perform predictions. Important channels contribute most to the prediction accuracy, as shown by Fig R.1 (b) in the attached PDF. In Fig R.1 (b), we can find Central sulcus contributes most to the prediction accuracy, followed by Frontal, Occipital and Parietal lobes, and plural available regions are generally better than single region. This aligns with the central sulcus's role in movement and its close anatomy relationship with Basal Ganglia, which is the primary brain region indicating Parkinson's disease [9], meaning that our model has successfully learned to pick up informations from brain regions that matters much more. Furthermore, Fig R.1 (a) shows how different routers are attending to different segment of the same signal (deeper red means that the router attends more to this patch). This demonstrates that other than capturing features at the 32Hz and 16Hz scaffolded by the patch length $L=8$ and $L=16$ for 256Hz EEG signals, the model is as well focusing on brain waves at 8~10Hz due to which the model is "downsampling" the tokens with attention scores in stripe patterns. This precisely aligns with the interval at which the EEG signals differs the most between healthy individuals and patients [10] and therefore is crucial in the diagnosis process of Parkinson's disease. In conclusion, our method is capable of focusing on important channels as well as time-frequency information which is well-supported by literatures and may give inspiration for future studies and applications.

Q2: The provided code could benefit from more comments.

A2: Thanks for raising this concern. We will update and add more comments to our code in our next version.

Q3: Does random seed only affect the initialization of model? Are the training/validation/test splits the same for all five runs?

A3: Yes. The random seed only affects the model's initialization, while the train-test splits remain consistent across the 5 runs. To satisfy the subject-independent setup, where the train-test split are based on subjects, we fixed the subjects used in each set. This means that the subjects included in the training, validation, and test sets remain the same, and only the model's initialization is randomized for each run.

Q4: What was your reasoning behind the different augmentation techniques used depending on the dataset?

A4: We did some preliminary analysis and evaluations to choose augmentation methods. Our goal is to select augmentation methods that do not change the label for classification while introducing some variance in the samples. In general, timestamp masking is the safest meth for classification tasks, followed by jittering. Timestamp masking involves randomly masking certain timestamps in the time domain without damaging the semantic information. Jittering, when applied with a low ratio of random noise, also preserves the semantic information in the raw data. We also experimented with frequency band masking but found that it sometimes negatively impacted performance. We believe this could be due to the semantic information in some datasets being concentrated in specific frequency bands, which this type of augmentation may disrupt.

Q5: Considering the societal impact of their model, steps should be taken to prevent the disclosing of demographic information in the dataset.

A5: Thank you for raising this concern. During data preprocessing, we manually removed the demographic information in the dataset and did not incorporate it into model training. We strongly agree that protecting sensitive information in the data is important when dealing with medical data. To further protect the demongraphic information, we may incorporate techniques such as differential privacy in future works.

Thank you for your suggestions to our paper! We are happy you interested in our design for experiments and carefully review our paper and code. Here are the detailed response for your questions and concerns.

Q1: The limitations should be discussed in the main paper, not appendix.

A1: Thanks for the suggestion. We will move the limiation to main paper in our revised version.

Q2: Baselines are all transformer-based methods. It would be interesting to compare with different architectures (e.g., try the Mamba model).

A2: Thanks for raising this concern. We add three more baselines that are not transformer-based for comparision: TCN, ModernTCN, and Mamba. The results are presented in R-Table 1 in the attached pdf file in general response. We evaluated our method and 6 baselines across the 3 new datasets and the previously used TDBrain dataset. Our method achieved the best F1 score on 3 datasets and the second-best on the remaining one.

Q3: The improvements over existing methods seems marginal. Some results are tied as they lie in the range of the standard deviation of the best model. It would be good to highlight these results as well.

A3: Thanks for raising this concern. PatchTST outperforms our method on the PTB-XL dataset by 0.6 in F1 score. As we observe from the ablation study, PTB-XL does not benefit from cross-channel patching too much; we guess the reason is the smaller number of channels(12) of PTB-XL. In other words, we believe our model is more suitable for datasets that demonstrate more channel correlations. We will discuss and analyze more on results in our revised version of paper.

Q4: The efficiency of the models should be benchmarked (e.g., training time).

A4: We have included a table (Table R.2) in the attached PDF that presents the runtime performance of several older baselines known for their strong performance, as well as the three new baselines, on the newly added dataset: FLAAP (13,123 samples, 10 classes). In general, CNN-based methods are fast, and the Mamba is extremely slow for unknown reasons. We guess this is because Mamba's new architecture does not match the GPU hardware optimization well. Our method achieves close runing time to other transformer-based methods.

Q5: Can the superior performance also extends to other vital signs and medical time-series prediction tasks? For example, PPG data and other multi-modal datasets.

A5: Yes, we believe our model is adaptable to various types of medical time series, particularly those with multi-channel correlations. For multi-modal learning of medical time series collected simultaneously (e.g., PPG, EEG, fNIRS, ECG), we see three potential research directions for applying our method: 1) Interpolating and Concatenating: Interpolating all modalities to the same length and vertically concatenating them into a multi-channel time series. 2) Individual Representation Learning: Using separate models to learn individual representations for each modality, then concatenating these representations for downstream tasks. 3) Granularity-based Modality Correspondence: Applying different granularities to correspond to different modalities during the learning process. We plan to explore these directions in future work.

Additionally, we train our method on a new dataset, MIMIC-PERform-AF, which uses PPG and ECG data to distinguish atrial fibrillation patients (20,400 samples, 2 classes). The results, presented in R-Table 1, show that our method outperforms the second-best result by approximately 3% in F1 score.

Dear Authors,

Thank you for addressing my concerns with additional material. However, a revised version of the paper must be provided to fully address the points in my and the other authors' reviews. Thank you.

Thank you for your suggestions on our paper! We are happy you feel our paper is well-structured and easy to follow. We respond to each of your questions and concerns. If you do not feel we have sufficiently justified a higher score, please let us know your further concerns and how to improve. Thank you again!

Q1: The idea of leveraging multi-granularity and multi-channel features has been studied in time series field.

A1: We agree that some methods utilized multi-granularity or multi-channel features in the time series domain, as we compared or described in the related work. However, we are the first to combine multi-granularity with cross-channel patching to model cross-channel information across different levels of granularity. In particular, we design an embedding method for medical time series data, where different channels represent signals from different regions of the same organs (brain, heart, etc.). Tight correlations among the channels are evident and valuable from a medical perspective[4][5][6]. Therefore, our approach is tailored to capture these correlations effectively with one-step data embedding. In contrast, as we mentioned in the paper (lines 107-110), existing patching methods in time series transformers, such as those inherited from PatchTST [1][2][3], rely on single-channel patching. These methods are designed for time series forecasting in tasks like weather and energy prediction, where different channels may represent entirely different meanings and, therefore, follow a channel-independent strategy. Thus, our method exhibits a significant deviation from the other methods and is a rational design for the specific datasets and challenges.

Q2: The paper should focus on the challenges of medical time series classification. What are those challenges, and how does Medformer resolve them?

A2:

1. Strong correlation between channels and features at a wide frequency range motivates our design of multi-granularity cross-channel patching: Medical time series data are collected from human subjects using sensors and electrodes, usually resulting in multi-channel and multi-modal data. As we discussed in our previous response, different channels in medical time series often represent activities in various regions of the same underlying organs, e.g., brain, heart, etc., from a medical perspective. Therefore, strong inter-channel correlations exist naturally [4][5][6]. Moreover, biomarkers in these signals appear at a wide range of frequencies with a difference of up to 1~2 magnitude [6]. To utilize these features, we design multi-granularity cross-channel patching that enables one-step data embedding, followed by a two-stage intra-inter granularity self-attention mechanism to exploit these features further.

2. The correlation of samples from the same subject encourages us to explore a subject-independent training setup. As we described in the preliminaries section(lines 124-135), for medical time series collected for disease diagnosis tasks, the ultimate goal is to predict the label of Subject instead of Sample to diagnose disease. In this case, samples from unseen subjects should not be utilized in the training to follow a real-world scenario. Many existing works do not consider subject ID information[7][8]. To better simulate real-world conditions, we categorize data splitting strategies into two setups: subject-dependent and subject-independent. For subject-dependent, samples are randomly split into training, validation, and test sets. For subject-independent, we split subjects into these three sets, ensuring that samples from the same subjects are exclusively contained within each set (See Figure 2). Tables 2 and 3 in the paper show how deceptively high results could be achieved with improper setup, leading to ineffective models in real-world applications.

Q3: Several new time series classification SOTA methods have been released, such as Time-LLM, ModernTCN, Lag-llama etc. These studies should be compared.

A3: Thank you for recommending these exceptional works. We will review them and incorporate them into the related work section in our upcoming revision. We have included experiments for ModernTCN in Table R.1 of the attached PDF. Despite our best efforts, we could not obtain results from the two LLM-based models, Time-LLM and Lag-Llama, as they are primarily designed for time series forecasting. Fine-tuning these pre-trained models for classification tasks would require more computational resources than currently available. However, to ensure a comprehensive evaluation of our model, we add two additional methods, TCN[11] and Mamba[13], also in Table R.1.

Q4: How does the Medformer perform in general time series classification?

A4: While our method is designed for medical time series using their characteristics, we believe it applies to general time series data, particularly multi-channel time series with intrinsic correlations among channels. To prove this, we add 3 new datasets to our experiments: FLAAP, UCI-HAR, and MIMIC-PERform-AF. The results are presented in R-Table 1, attached to the general response PDF. We evaluated our method and 6 baselines across the 3 new datasets and the previously used TDBrain dataset. Our method achieved the best F1 score on 3 datasets and the second-best on the remaining one.

Q5: Medformer's results are not superior in several metrics.

A5: PatchTST outperforms our method on the PTB-XL dataset by 0.6 in F1 score. Based on the ablation study presented in Table 4, cross-channel patching does not provide as much benefit as other datasets. We guess that this might caused by the smaller number of channels (12) in PTB-XL and weak channel correlation, which limit the advantages of our cross-channel patching. However, we believe the other mechanism, such as intra-inter granularity self-attention, can be combined with PatchTST to improve the result on PTB-XL.

Reviewer vj5L,

Thank you for your service in reviewing for NeurIPS! Your work is not done, however, as part of a reviewer's responsibility is to engage with the authors during the discussion period.

The authors of Submission 7556 (Medformer) have provided an extensive response to your review. Please read it and indicate the extend to which it addresses your concerns. Please indicate clearly which of the issues you raised are addressed and which are not, such that the authors have a final chance to reply.

There is only one day left in the discussion period, so please do this ASAP!

The AC for Submission 7556

Dear reviewer vj5L:

We would like to kindly remind you that the author-reviewer discussion will end in 1 day. Could you take a look at our rebuttal and let us know if our rebuttal has addressed your concerns? As our paper is currently on the borderline, your input will be greatly appreciated.

Best regards

Thank you for your feedback and concerns regarding our paper! We appreciate your thoughtful questions. Here are the reponses to your question and concern. If you feel our response does not fully justify a higher score, please let us know how we can further improve our work. Thank you again for your consideration.

Q1: Extension to general time series outside the medical domain.

A1: Thank you for raising this concern. Our answer contains two parts.

1. We agree our method is designed for medical time series due to the inherent correlations among channels from a medical perspective[4][5][6]. However, we believe our method has the potential to be extended to other domains as well. Specifically, our approach is particularly well-suited for multi-channel time series data where theoretical correlations exist among different channels. Our multi-granularity cross-channel patching effectively embeds these channel correlations in a single step. Potential domains for application include human vital signs for health monitoring and time series data collected from mobile sensors for human activity recognition.

2. To further evaluate the generalizability of our method, we add three new datasets, two human activities recognition datasets FLAAP(13123 samples, 10 classes) and UCI-HAR(10299 samples, 6 classes), and one multi-modal vital signs(PPG and ECG) dataset MIMIC-PERform-AF(20400 samples, 2 classes) for atrial fibrillation classification. See R-Table 1 for the results in the pdf file attached to the general response. In this table, we compare with three old baselines with good performance so far, Crossformer, Reformer, and Transformer, and three new baselines TCN, ModernTCN, and Mamba. We evaluate our method and the six baselines on three new datasets and one old dataset, TDBrain. Our method demonstrates top-1 Accuracy and F1 score in three datasets and second-best in the remaining one.