EEGPT: Unleashing the Potential of EEG Generalist Foundation Model by Autoregressive Pre-training

1.Innovation: The proposed pretraining method does not extend beyond the two tasks used in Bert [1] — mask-based reconstruction and next sentence prediction. While Bert employs both tasks simultaneously in training language models, and LaBraM[2] utilizes the mask-based reconstruction approach, this paper explores the other method without attempting to combine both approaches, limiting its innovation in large model pretraining methods.
2.Method design: The construction of large models generally focuses more on unsupervised pretraining methods rather than the benefits of multi-task fine-tuning across different datasets. This requires the assumption of sufficient annotated data, and the high cost of annotation may make this approach impractical for real-world tasks.
3.Clarity of description: When discussing the unification of different dataset formats for input construction, the paper mentions organizing inputs by channel and adding a special channel identifier for each sample. However, significant dimensional discrepancies among different datasets and how to mitigate these discrepancies through unified input construction (possibly using LaBraM’s method to construct dense inputs) are not clearly explained. Further clarification is recommended.

[1]Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. 2019.
[2]Wei-Bang Jiang, Li-Ming Zhao, and Bao-Liang Lu. Large brain model for learning generic representations with tremendous eeg data in bci. arXiv preprint arXiv:2405.18765, 2024.

Tokenization strategy

My biggest concern about the submission is the lack of investigations on EEG tokenization. The main text does not provide any information about EEG pre-processing and harmonization, leaving the reader puzzled about the dimensions $T$ and $C$ as defined in section 2.1 , and whether the data were harmonized (e.g., to a common sampling rate). After some digging, I found a short paragraph on data preprocessing in the Appendix on page 16 (but no reference to it in the main text!). This paragraph contains important information to understand the approach of the method and should, therefore, be move to the main text.

Without any justification or reference to prior work, the authors decided to use substantially overlapping short temporal windows (i.e., 1 second) to segment slightly longer windows of EEG time-series data (i.e., 4 seconds) into 25 tokens (i.e., T = 25), each containing 1 second of data at a rate of 256 Hz (i.e., C = 256). Since auto-regressive pre-training is one of the main contributions of this paper, the authors should provide additional motivation for the particular choice of hyper-parameters and ideally run ablation studies that investigate different hyper-parameter choices (e.g., length of the long and short windows as well as the overlap).

Clarity of multi-electrode and multi-task fine-tuning approach

The authors describe their multi-electrode integration approach as a graph that is task-specific. Unfortunately, the provided description is insufficient. A graph typically comprises vertices (or nodes) and edges. The author's merely define the node features but do not clearly introduce how they determine edges and their values (based on equation (9), I assume that they use binary edge-weights based on the availability of an electrode in a sample $z_j$). Additionally, in equation (8) the authors state that $\mathrm{diag}(z_j)$ converts $z_j$ into a matrix. However, $z_j$, defined in equation (7), is already a matrix. Overall, the clarity of the this section would benefit a lot from proper function definitions that include the function's image and domain.

Organization

The authors spend a considerable amount of text on repeatedly highlighting their perspective of the submission's contributions. I think that the authors should rather expand on the methods description and experimental results.

Major

• The evaluation method is problematic and does not support the main claim of the article, which is that the authors introduce a “generalist EEG foundation model”. As stated in lines 660-664, the authors “adopt a cross-subject paradigm”, and the validation and test sets contain different subjects but from the same datasets as for training. Therefore, the model does not realise a cross-task transfer but rather a simple cross-subject transfer learning scenario (augmented with examples from other tasks). The variability between different subjects is not comparable to that of different datasets or even different decoding tasks. In this article, the model was trained on subjects doing the exact same tasks from the exact same datasets as during testing. Therefore, it is impossible to predict from these results how the model will behave on new tasks from new datasets.
• The authors make multiple strong claims (all starting with “the first…”) which are either overstatements, if not wrong. The presentation of this work raises concerns about its scholarly rigour and gives the impression of being more like a product promotion than a scientific publication. The claims are the following:
  • “first electrode-wise modeling strategy”. See for example:
    • Yang et al. (2023) http://arxiv.org/abs/2305.10351
    • Guetschel et al. (2024) https://doi.org/10.3217/978-3-99161-014-4-003
    • Li et al. (2024) https://doi.org/10.1109/TNSRE.2024.3357863
  • “first autoregressive model”. See Banville et al. (2021) https://www.doi.org/10.1088/1741-2552/abca18. Moreover, multiple articles already explored masking-based SSL strategies, which are not strictly speaking “autoregressive” but still very similar. See for example:
    • Foumani et al. (2024) https://doi.org/10.1145/3637528.3671600
    • Yang et al. (2023) http://arxiv.org/abs/2305.10351
  • “first generalist EEG model for multi-task compatibility and synergism”. See Yang et al. (2023) http://arxiv.org/abs/2305.10351. moreover, multiple works already explored cross-task transfer learning in BCI:
    • Aristimunha et al. (2023) https://arxiv.org/abs/2308.02408
    • Guetschel et al. (2023) https://arxiv.org/abs/2311.16109
• Some results appear significantly below the state-of-the-art. Some variability might be explained by differences in the evaluation settings but the differences reported (10%) are non-negligible. Unfortunately, I am not familiar with all datasets but here are those I know well:
  • MIBCI. Zhao et al. (2020) obtained 76.5% accuracy in a similar cross-subject configuration, which is 9.3% above the reported score and 13.2% above the best specialist model https://doi.org/10.1155/2020/7285057
  • Seed-IV. Li et al. (2018) reported that a simple SVM (one of their baselines) obtained 52.8% accuracy in a similar cross-subject transfer setting, which is 11.5% below the reported score and 18.1% below the best specialist model https://doi.org/10.1007/978-3-030-04221-9_36

Minor

• The name EEGPT is probably not the best choice as two articles already used that name for their architectures:
  • https://neurips.cc/virtual/2024/poster/93793
  • https://arxiv.org/abs/2401.18006
• Figure 1 is very difficult to read. The axes scale is defined by the blue curve therefore, the tickmarks land on random values. I understand that the scores on the different datasets are not comparable together but you should at least scale the axes such that the tickmarks land on integer values, not floats with two decimals. Also, it would help the reader to add the range of each axis next to its name.
• line 86: the claim that MAE “have inevitable limitations in capturing the sequential and temporal dependencies” is not motivated. Research actually showed that they can. See Chien et al. (2022) http://arxiv.org/abs/2211.02625.
• The choice of motor imagery datasets seems rather arbitrary. Cho 2017 and BCI Competition IV 1 are not commonly used for benchmarking. Instead, I would have recommended:
  • Very common benchmark: dataset B from 2008 BCI competition http://moabb.neurotechx.com/docs/generated/moabb.datasets.BNCI2014_004.html
  • More recent and one of the largest motor imagery datasets: Stieger et al. (2021) http://moabb.neurotechx.com/docs/generated/moabb.datasets.Stieger2021.html

Additional references should support the reported results, as certain datasets appear to have higher state-of-the-art (SOTA) values than those presented. Sharing anonymized code would enhance reproducibility and allow for broader validation of the model. Lastly, an in-depth discussion on how the model captures covariances between variables is crucial, given its importance in the EEG literature. This would provide clarity on EEGPT’s handling of multivariate dependencies.