Are Large Brainwave Foundation Models Capable Yet ? Insights from Fine-Tuning

We would like to thank the reviewer for their thoughtful review and valuable feedback. As mentioned in our response to Reviewer QMtY, we have decided also to extend our analysis results to capture one more popular BCI paradigm, namely Motor. Therefore, we added a new movement benchmark based on the High Gamma dataset (R T Schirrmeister et al, Deep learning with convolutional neural networks for eeg decoding and visualization. Human brain mapping, 38(11):5391–5420, 2017). Please see our response to Reviewer QMtY for the updated results Table. This extends our thorough analysis even further as mentioned by the current Reviewer bhSh.

The performance difference between NeuroGPT (the best foundation model in the new analysis) and EEG-Inception (the best baseline network) is 1.2% (improvement compared to the previous 0.5%) but with a considerable increase in the number of trainable parameters and larger std among folds. As highlighted by the reviewer, although we demonstrate 10‐fold cross‐validation performance, the original manuscript does not show whether that small improvement margin of 1.2% is statistically significant or not. For that reason, we conducted paired-t tests between EEGInception (the best baseline network) and all examined foundation models. The results are demonstrated below, showing a clear statistical significance for NeuroGPT in many benchmark tasks:

Table: P-values of paired-t tests between EEGInception and finetuned foundation models. Bold values indicate statistically significant results $(p < 0.05)$.

As suggested by the reviewer, we conducted another study: fixed LBMs but learnable classifiers. The results are shown in the table below:

Table 2: Classification accuracy of foundation models where all parameters except classification heads are frozen. Each model was trained for 20 epochs with 10-fold cross-validation. Bold values indicate the best performance per task or overall.

From the results above, training just the classification yields models that lack behind baselines by a large margin almost 8-10% (comparing with the results in our response to reviewer QMtY). This further demonstrates that current brainwave foundation models lack essential elements to fully capture the diverse nature of EEG and largely outperform all current state-of-the-art baselines in various tasks with minimum required fine-tuning. This constitutes full-model fine-tuning necessary and in turn makes PEFT techniques like LoRA extremely valuable.

The classifiers in both foundation models were designed based on the implementations of the original foundation models papers (LaBraM and Neuro-GPT). As mentioned in our response to Reviwer 9Has, both foundation models (LaBraM and Neuro-GPT) have in their pre-training datasets paradigms that include motor-, ERP-, sleep- and eyes-related tasks (not the specific datasets chosen for the downstream tasks). Interestingly, in the memory task—which was not explicitly included in the pre-training datasets—baseline models slightly outperform foundation models.

We thank the reviewer for the additional references provided. In our studies, we included the two state-of-the-art open-source EEG foundation models of that time. The trained weights of CBraMod and NeuroLM were released close to the ICML submission deadline. Including these models in our analysis is not possible due to rebuttal's time constraints. But we will definitely acknowledge these recent works in the background section of our revised manuscript.

In the camera-ready version of the manuscript, we will incorporate the paired-t test analysis, the frozen foundation models analysis and the aforementioned discussion points. We sincerely thank the reviewer for their valuable comments, which have helped us to improve our work.

We would like to thank the reviewer for their thoughtful review and valuable feedback. As mentioned in our response to Reviewer QMtY, we have decided also to extend our analysis results to capture one more popular BCI paradigm, namely Motor. Therefore, we added a new movement benchmark based on the High Gamma dataset (Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human brain mapping, 38(11):5391–5420, 2017). Please see our response to Reviewer QMtY for the updated results Table. This extends our thorough analysis even further as mentioned by the current Reviewer MQLQ.

The performance difference between NeuroGPT (the best foundation model in the new analysis) and EEG-Inception (the best baseline network) is 1.2% (improvement compared to the previous 0.5%) but with a considerable increase in the number of trainable parameters and larger std among folds. As highlighted by the reviewer, although we demonstrate 10‐fold cross‐validation performance, the original manuscript does not show whether that small improvement margin of 1.2% is statistically significant or not. For that reason, we conducted paired-t tests between EEGInception (the best baseline network) and all examined foundation models. The results are demonstrated below, showing a clear statistical significance for NeuroGPT in many benchmark tasks:

P-values of paired-t tests between EEGInception and finetuned foundation models. Bold values indicate statistically significant result $(p < 0.05)$

We agree with the reviewer "brain foundation models" could refer to models on fMRI as well therefore we have used the term "Brainwave Foundation Models (LBMs)", clearly hinting the EEG modality.

In the camera-ready version of the manuscript, we will incorporate the paired-t test analysis. We sincerely thank the reviewer for their valuable comments, which have helped us to improve our work.

We would like to thank the reviewer for their thoughtful review and valuable feedback. As mentioned in our response to Reviewer QMtY, we have decided also to extend our analysis results to capture one more popular BCI paradigm, namely Motor. Therefore, we added a new movement benchmark based on the High Gamma dataset (Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human brain mapping, 38(11):5391–5420, 2017). Please see our response to Reviewer QMtY for the updated results Table. This extends our thorough analysis even further as highlighted by the current Reviewer 9Has.

As highlighted by the reviewer, explainability is crucial in BCI applications. Therefore, developing interpretable AI models is essential. In recent years, many research papers have developed interpretable EEG models with strong performance. As we head toward large brainwave foundation models, maintaining this requirement remains critical. While Neuro-GPT follows a black-box approach, LaBraM incorporates a neural codebook method, enhancing model interpretability, as discussed in the background section of the manuscript.

As shown in the analysis Table in our response to Reviewer QMtY, foundation models slightly outperform baseline models by an average of 1.2%. However, as the reviewer rightly points out, a closer examination of individual tasks and the pre-training datasets used by these models is worth mentioning. Both foundation models (LaBraM and Neuro-GPT) have in their pre-training datasets paradigms that include motor-, ERP-, sleep- and eyes-related tasks. From the results, we observe that foundation models achieve comparable performance to baseline models in ERP, sleep, and eyes tasks. NeuroGPT significantly outperforms baseline models in motor. Interestingly, in the memory task—which was not explicitly included in the pre-training datasets—baseline models slightly outperform foundation models. This evidence further strengthens our claim that these models have yet to reach their full potential - particularly in achieving significant generalization across new tasks. By integrating domain-specific knowledge, such as leveraging various EEG modalities, and employing tailored training strategies, like brain-inspired masking techniques, these models could fully capture the diverse nature of EEG and largely outperform all current state-of-the-art baselines in various tasks with minimum required fine-tuning.

In the camera-ready version of the manuscript, we will incorporate the aforementioned discussion points. We sincerely thank the reviewer for their valuable comments, which have helped us to improve our work.

We would like to thank the reviewer for their thoughtful review and valuable feedback. As the reviewer highlights we have meticulously selected various benchmark BCI paradigms datasets, reflecting the current state-of-the-art within the EEG community. We decided also to extend these results to capture one more popular BCI paradigm, namely Motor. Therefore, we added a new movement benchmark based on the High Gamma dataset (Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human brain mapping, 38(11):5391–5420, 2017). The updated results are presented below:

Table: Classification accuracy of finetuned foundation models and standard deep learning architectures, reported as mean (std). Each trained/finetuned for 20 epochs with 10 fold cross-validation. Bold values indicate best performance per task or overall, italic values indicate next best performance.

The performance difference between NeuroGPT (the best foundation model in the new analysis) and EEG-Inception (the best baseline network) is 1.2% (improvement compared to the previous 0.5%) but with a considerable increase in the number of trainable parameters and larger std among folds.

In terms of a more pronounced perspective on the future outlook for LBMs, we strongly believe that this extensive study highlights critical considerations for the research community. In the camera-ready version of the manuscript, we could also add that we believe the future of Large Brainwave Foundation Models (LBMs) should go beyond merely adopting transfer techniques from other domains. Instead, they should integrate domain-specific knowledge—such as leveraging various EEG modalities—and employ tailored training strategies, like brain-inspired masking techniques. These are essential elements to fully capture the diverse nature of EEG and build an efficient and effective LBM that will largely outperform all current state-of-the-art baselines in various tasks with minimum required fine-tuning.

In the camera-ready version of the manuscript, we will incorporate the aforementioned discussion points. We sincerely thank the reviewer for their valuable comments, which have helped us to improve our work.