MTEEG: A Multi-Task Learning Framework for Enhanced Electroencephalography Analysis Using Low-Rank Adaptation

Reviewer muVQ

We appreciate reviewer muVQ's insightful comments. Please see below for the point-by-point responses.

[W1] The novelty of the paper is somewhat limited. Based on the experimental results, I question the necessity of multi-task learning in this context.

Following reviewer's advice, we conducted experiments on single-task fine-tuning with LoRA (with LaBraM-Base backbone), the results of which are shown in the table below. To stay consistent with the experimental settings in the main paper, we repeat the experiments with three different random seeds (0, 5, 10) and report the mean and std values.

From the table, we can see that the performance of single-task fine-tuning with LoRA is pretty similar to that of single-task full fine-tuning (the LaBraM-Base rows in Table 2-4), and multi-task fine-tuning is generally comparable to single-task fine-tuning, i.e, better on four datasets and worse on the other two, which, in the reviewer's opinion, indicates that multi-task fine-tuning lacks necessity. However, we would like to advocate multi-task fine-tuning for the following reasons.

• The performance of multi-task fine-tuning is actually good. If we look at the research works on multi-task learning in other domains, such as image or language processing, it's very common that a newly proposed method is better than the baselines on some tasks but worse on some others, so researchers often use the average performance across all the tasks as an "ultimate metric" to demonstrate their methods' superiority. We didn't do this because we follow previous EEG works to use different performance metrics for different tasks (datasets), so it doesn't make much sense to take the average of values with different meanings. That being said, balanced accuracy is a common metric that we use for all the tasks, so we report the average values of it under three different settings in the table below. As we can see, the performance of multi-task fine-tuning is consistently better than the single-task counterparts.

• Multi-task fine-tuning is beneficial for real-world deployment. Setting the performance aside, multi-task learning is still meaningful because of its efficiency. It saves space, computation, time, and in real-life applications, it's convenient to have one unified system that can handle all the tasks simultaneously. As an example, researchers in the computer vision community used to develop separate models for classification, detection, and segmentation tasks but nowadays most of them have turned their attention to multi-task models, even though the performance of a multi-task model is not always better than a task-specific model. Currently, EEG recognition is still dominated by task-specific models, so we would like to make the first try. Hopefully, more researchers will pay attention to multi-task models in the future and develop models that are both powerful and efficient enough to be deployed in real-life BCI devices.

In addition, we also conducted experiments with TCN and Medformer on TUEV and SEED-V, as suggested by the reviewer. As shown in the table below, TCN performs really well on TUEV, but both models are not good at emotion recognition on SEED-V.

[Q1] Could you provide more details about your data split? I'd like to know whether you are using the subject-dependent or subject-independent setup

We totally agree with the reviewer on that a subject-dependent setup focuses too much on the performance numbers rather than applicability in real-world scenarios, so in this work, we use a subject-independent setup for all the datasets except for SEED-V. We chose to use a subject-dependent setup for SEED-V simply because we wanted to stay consistent with previous works, especially LaBraM, which is the basis of this work and a major baseline that we compare with.

We thank reviewer 2WaD for the thorough review and valuable suggestions. Please see below for the point-by-point responses.

[W1] The contribution appears limited

We acknowledge that the core techniques employed in MTEEG, such as pre-trained models and LoRA, are not novel, but still would like to highlight the significant contributions of our work as follows:

• Despite its importance, multi-task learning (MTL) has remained unexplored in the field of EEG recognition. Although some previous works [1][2][3] claim to have incorporated MTL into their frameworks, the term “task” in their works refer to the hand-crafted pretext tasks that are used to improve optimization on a single dataset, while in our work, the term “task” refers to the actual downstream tasks. Therefore, MTEEG is the first framework which enables multi-task capability by training on a wide range of different datasets simultaneously.
• The lightweight design of MTEEG ensures both space and time efficiency, which will greatly benefit the deployment of pre-trained EEG models in real-world scenarios. Moreover, MTEEG achieves this efficiency without sacrificing performance, as proven by the fact that it performs comparably with LaBraM (fine-tuned in single-task settings).

[W2] The paper benchmarks the proposed MTEEG framework against older multi-task learning methods like HPS, MMoE, and CGC, which, while relevant, are not cutting-edge.

We chose these as the multi-task baselines in our work for the following reasons.

• Since MTEEG is a pioneering work in exploring multi-task learning for EEG recognition, there’s currently no public method in this domain to compare with, so we decided to integrate LaBraM into some general MTL frameworks to set up the multi-task baselines. Although some more recently proposed methods have demonstrated good performance on image datasets, they often incorporate architecture-specific designs, making it unclear how they could be effectively consolidated with LaBraM for applications in the EEG domain. In contrast, while methods like HPS, MMoE and CGC are old, they can be seamlessly employed to any backbone networks, including LaBraM, without any architecture-specific designs.
• The performance of these methods are still good enough that many recent papers [4][5] include them as the baselines.
• These methods have been re-implemented by a group of researchers and integrated into the LibMTL library. By using this library, we can ensure fair comparison and reproducibility.

[W3] For a more thorough assessment, it would be valuable to demonstrate if techniques like Adapters, Prefix-tuning, or Prompt Tuning could seamlessly replace LoRA or be integrated into the MTEEG framework.

We greatly appreciate this valuable suggestion and will add these baselines to strengthen our work.

[Q1] Have you considered a simpler approach, that is using a task-specific classification head for each dataset while fine-tuning the entire pre-trained LaBraM model?

We thank the reviewer for pointing this out. Yes, we have considered this simpler approach, and this is exactly what HPS (Hard Parameter Sharing) does. In Section 4.3 of the paper, we attribute the performance gap between HPS and single-task LaBraM-Base to the conflicts between different datasets, since the architectures of these approaches are almost identical, except that HPS has more classification heads.

References

[1] Autthasan, Phairot, et al. "MIN2Net: End-to-end multi-task learning for subject-independent motor imagery EEG classification." IEEE Transactions on Biomedical Engineering 69.6 (2021): 2105-2118.

[2] Ditthapron, Apiwat, et al. "Universal joint feature extraction for P300 EEG classification using multi-task autoencoder." IEEE access 7 (2019): 68415-68428.

[3] Li, Yang, et al. "GMSS: Graph-based multi-task self-supervised learning for EEG emotion recognition." IEEE Transactions on Affective Computing 14.3 (2022): 2512-2525.

[4] Chang, Jianxin, et al. "Pepnet: Parameter and embedding personalized network for infusing with personalized prior information." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.

[5] Cao, Jiangxia, et al. "Towards universal cross-domain recommendation." Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining. 2023.

We are grateful for reviewer zWj6’s careful review and insightful comments. Please see below for the point-by-point responses.

[W1] The paper lacks clarity and self-containment, making it difficult for readers to fully grasp the research objectives and methodologies.

We thank the reviewer for pointing this out, which will help us improve the clarity of our paper. The fundamental idea behind this work is that while current EEG pre-training models are powerful, they require full fine-tuning to perform well on a downstream task. In a clinical environment, we may need many types of information (e.g. seizure, sleep, emotion, etc.) to be simultaneously decoded from the patient’s brain signal to comprehensively evaluate the patient’s health condition. In this case, a straightforward solution would be to replicate and fine-tune a pre-trained model multiple times, once for each type of information (task). However, doing this will introduce significant space (to store all the replicas) and computation/time (to optimize all the replicas) overhead. Therefore, the primary question we want to ask here is: Can we just have one unified model that can handle all the tasks simultaneously? To achieve this, the model needs to be fine-tuned simultaneously on multiple different datasets, which poses challenges since the heterogeneity can cause conflicts in the optimization directions. A direct consequence of the conflict issue is a significant drop in downstream performance, which is shown by the performance gap between HPS and LaBraM-Base in Table 2, 3, and 4. HPS (hard parameter sharing) here refers to the naïve implementation where we attach task-specific classification heads to a pre-trained LaBraM-Base and directly fine-tune the whole model on all datasets. To mitigate the conflict issue, we apply the task-specific LoRAs to isolate the parameter space, while keeping some modules task-independent so that different tasks can still benefit from some common knowledge. Overall, 1) The proposed MTEEG is more space and computation/time efficient, which addresses the problem of fine-tuning a pre-trained model multiple times as described above. 2) MTEEG achieves this efficiency without sacrificing performance, as proven by the fact that it performs comparably with LaBraM (fine-tuned in single-task settings).

[W2] The paper appears to be a combination of existing methodologies and lacks a significant novel contribution to the field

We acknowledge that the core techniques employed in MTEEG, such as pre-trained models and LoRA, are not novel, but still would like to highlight the significant contributions of our work as follows:

• Despite its importance, multi-task learning (MTL) has remained unexplored in the field of EEG recognition. Although some previous works [1][2][3] claim to have incorporated MTL into their frameworks, the term “task” in their works refer to the hand-crafted pretext tasks that are used to improve optimization on a single dataset, while in our work, the term “task” refers to the actual downstream tasks. Therefore, MTEEG is the first framework which enables multi-task capability by training on a wide range of different datasets simultaneously.
• As mentioned in our response to the previous question, the lightweight design of MTEEG ensures both space and time efficiency, which will greatly benefit the deployment of pre-trained EEG models in real-world scenarios.

[W3] The authors should clarify how their work differs from transfer learning

We thank the reviewer for pointing out the need for this clarification. Transfer learning is a general concept which, to our knowledge, refers to the process of adapting a pre-trained model to a given target task. In this sense, our work is a special form of transfer learning. There are two major differences between MTEEG and a standard/typical implementation of transfer learning.

• Transfer learning typically involves only one downstream dataset at a time, but MTEEG fine-tunes a pre-trained model on multiple datasets concurrently. This poses a unique challenge regarding the conflicts between different datasets, so the model needs to be carefully designed to reach a balance between what should be shared versus what should be disentangled.
• As the reviewer mentioned, a common practice in transfer learning is to freeze some layers and train the others for a specific task. However, current EEG pre-training models require full fine-tuning to achieve good downstream performance. Thus, MTEEG does not “freeze” any layer conceptually, it just utilizes the LoRA modules to approximate the changes of all the parameters.

[Q1] The authors should clarify how their work differs from transfer learning

This question turns out to be a duplicate of W3. Please refer to our response above.

[Q2] The authors should provide a clear explanation of how they resampled the EEG signals to 200 Hz

We agree with the reviewer on that the resampling strategy should be consistent across datasets. In this work, the MNE-Python [4] library is used for all the preprocessing work, including resampling. We called the resample function with default parameters, that is, resample(sfreq, *, npad='auto', window='auto', stim_picks=None, n_jobs=None, events=None, pad='auto', method='fft', verbose=None). Looking into the source code of the library, we found that the resample function first pads the input so that the sequence length is a power of two, then creates a boxcar window, and calls scipy.signal.resample() function to perform resampling with the Fourier-based method.

References

[1] Autthasan, Phairot, et al. "MIN2Net: End-to-end multi-task learning for subject-independent motor imagery EEG classification." IEEE Transactions on Biomedical Engineering 69.6 (2021): 2105-2118.

[2] Ditthapron, Apiwat, et al. "Universal joint feature extraction for P300 EEG classification using multi-task autoencoder." IEEE access 7 (2019): 68415-68428.

[3] Li, Yang, et al. "GMSS: Graph-based multi-task self-supervised learning for EEG emotion recognition." IEEE Transactions on Affective Computing 14.3 (2022): 2512-2525.

[4] Gramfort, Alexandre, et al. "MEG and EEG data analysis with MNE-Python." Frontiers in Neuroinformatics 7 (2013): 267.

[Q1] Its unclear to me what the central hypothesis and question of the study is. Is it simply about saving time during fine-tuning or inference when all downstream tasks are known apriori?

Yes, time, space, and computation efficiency are valuable features that will greatly facilitate real-world deployment. Moreover, MTEEG achieves this efficiency without sacrificing performance.

[Q2] Typo in equation at line 170: Do you mean C_p instead of C_k?

Yes, it should be $C_p$ instead of $C_k$. We thank the reviewer for pointing out this typo and will correct it in the revision.

[Q3] What exactly happens in line 174? Do you mean temporal and spatial positional encodings?

Yes, temporal and spatial embeddings in line 174 refer to the positional encodings. We omitted some details here to save space since the positional encodings are carried over from the original LaBraM paper. Sorry for the confusion.

[Q4] p refers to tasks (line 208, figure 2) and datasets (line 154) interchangeably, which is quite confusing. You can explain/justify the equality at the beginning and use either meaning consistently throughout the paper.

We are grateful for the reviewer's valuable suggestion and will add explanation for this in the revision.

[Q5] Is the variability shown in Figures 4 and 5 statistically significant?

This question seems to be a duplicate of [W2]. Please refer to our response above.

[Q6] Is the parameter efficiency of MTEEG only due to the low-rank adaptor module, or are there other differences b/w LaBraM-base and MTEEG-base?

Yes, the only structural difference between these two is the LoRA modules.

[Q7] Consider adding experiments that can show the heterogeneous conflict issue in multi-task fine-tuning

We appreciate this really nice suggestion and will definitely consider it.

References

[1] Jiang, Wei-Bang, Li-Ming Zhao, and Bao-Liang Lu. "Large brain model for learning generic representations with tremendous EEG data in BCI." arXiv preprint arXiv:2405.18765 (2024).

[2] Schirrmeister, Robin Tibor, et al. "Deep learning with convolutional neural networks for EEG decoding and visualization." Human brain mapping 38.11 (2017): 5391-5420.

[3] https://openreview.net/forum?id=QzTpTRVtrP

[4] Yang, Chaoqi, M. Westover, and Jimeng Sun. "Biot: Biosignal transformer for cross-data learning in the wild." Advances in Neural Information Processing Systems 36 (2024).

[5] Cui, Wenhui, et al. "Neuro-GPT: Towards A Foundation Model For EEG." 2024 IEEE International Symposium on Biomedical Imaging (ISBI). IEEE, 2024.

[6] Yuan, Zhizhang, et al. "Brant-2: Foundation Model for Brain Signals." arXiv preprint arXiv:2402.10251 (2024).

We thank reviewer oDYw for providing a thorough review with constructive comments. Please see below for the point-by-point responses.

[W1] Preprocessing of the EEG signal needs improvement.

We thank the reviewer for providing the valuable suggestions. The reason why we chose the current preprocessing strategy is two-fold. Firstly, the same strategy has been applied in LaBraM[1], which is the framework that we build our method upon and compare our method to. We would like to stay consistent with LaBraM so that it makes more sense to attribute the performance difference to our core contribution (multi-task design) instead of other aspects such as preprocessing and data split. Secondly, since we are dealing with large-scale heterogeneous data, we are inclined to be conservative by only applying minimal preprocessing. While we agree with the reviewer on that anything beyond 25Hz (45Hz at most) is usually considered high-frequency noise, there are studies suggesting that higher frequencies could be useful [2]. It’s hard to tell what the optimal preprocessing strategy is for each of the datasets, so we decided to not filter out that much information and let the model itself determine what is important. Regarding the notch filter, we acknowledge it’s our oversight to carry this over. While the original authors of LaBraM think this is not that big of an issue [3], we thank the reviewer for pointing it out and helping us improve the quality of our work.

[W2] Reliability of results

We wonder if the reviewer could elaborate more on what type of variability is more important to report in the result section. To our knowledge, it seems common [1][4][5] to report the variability of performance metrics based on different random initializations of the model (with different random seeds) to ensure that the reported performance gain or loss is not simply due to randomness.

We thank the reviewer for highlighting the need for significance testing to support the ablation results. We performed t-test on the results in Figure 4 and 5 and show the p-values below. The three values in each cell correspond to the three performance metrics respectively.

[W3] The problem/motivation/hypothesis and real-world need/utility are unclear to me

We greatly appreciate the reviewer for helping to clarify what the actual need is in future clinical environments - having multiple EEG models/tasks "active" at the same time rather than switch from one model/task to another, so we have adjusted the manuscript accordingly to make this clear and hopefully avoid confusion in the future.

Regarding the second point, i.e., the time/computation overhead of fine-tuning a separate backbone for each task of interest. While this overhead can be negligible if the dataset and the model are both small enough, we argue that this will not be the case in the future, especially with the rapid advancements of neural signal collection and AI techniques. As a reference, the TUAB dataset used in this current study contains 409,455 10-second samples, and it takes 3:20:27 to fine-tune a LaBram-Base (5.8M) on it for 50 epochs with a batch size of 256 using two NVIDIA A100 GPUs. People may have different perspectives, but we already would not consider this time/space/computation cost as negligible, not to say that works in this field have recently scaled EEG pre-training models up to have billions of parameters [6].

[W4] Overall presentation can be substantially improved

We thank the reviewer for these valuable tips and will make changes accordingly in the revision.