Thank you for your review and comments. We appreciate your recognition of the value of our theoretical analysis and the usefulness of our experiments.

Clarification of differentiation to prior work on data augmentations

The point we were trying to make is not that others have not looked at augmentations in the context of limited data. Our work looks at two different dimensions of data scarcity: the number of participants, and the amount of data available for each participant. In contrast, the only previous piece of work systematically investigating the impact of augmentations in EEG has only considered a single dimension encapsulating the total amount of data available.

Moreover, we would like to emphasise that we provide additional reasons why our work is different in lines 85-89: “Furthermore, our experiments were conducted on much larger datasets with over 1,000 participants each, allowing us to study the impact of augmentations across a broader range of data regimes. Finally, their experiments were limited to a single model per task, whereas we benchmarked all augmentations with different model architectures for each task and observed differential effects.”

Representativeness of selected model for self-supervised learning

We will rephrase this claim in the manuscript to make our intentions clearer. LaBraM provides an example of a state-of-the-art method for self-supervised pre-training in EEG-based machine learning.

We would like to remark that the use of only one SSL method does not impact our analysis of research question Q4. Our research question is stated as follows: “Q4: Can self-supervised pre-training effectively increase performance across different data regimes, including limited participant counts or data per participant?” This does not require us to show that all self-supervised learning methods increase performance. We can answer this question by showing that at least one self-supervised learning approach works.

Evidence for claim in line 239 (doubling participants would yield “roughly 1%” improvement)

This claim refers to the observations stated earlier in the paragraph. Specifically, we refer to normality prediction on the TUAB dataset in Figure 2, where extrapolating the curve after 1600 participants, we would expect roughly a 1% improvement if the participant count was doubled for the TCN or mAtt model. The same is already observable (so we don’t have to extrapolate) when moving from 800 to 1600 participants. Finally, this can also be seen by comparing the top 2 rows in any of the plots in Figure 3. We will clarify this in the manuscript.

Motivation for selected augmentations

First we would like to clarify that Gaussian noise and cropping are not augmentations studied in our work. We studied AmplitudeScaling, FrequencyShift, and PhaseRandomization.

As discussed in the related works section, our selection was informed by the systematic evaluation of EEG augmentations conducted by Rommel et al. In their benchmarks on two tasks (sleep staging and BCI motor imagery), PhaseRandomization (termed FTSurrogate) consistently emerged as the most effective augmentation. FrequencyShift showed small improvements when data was more limited. Inclusion of the FrequencyShift augmentation was further motivated, because differences in peak frequencies have been shown to be associated with demographic variables like age and sex [1, 2, 3], making it an interesting augmentation to study in the context of participant diversity. In addition, we also studied AmplitudeScaling, an augmentation introduced by Mohsenvand et al. in the context of a contrastive learning approach, but not evaluated in the review by Rommel et al.

Importantly, Rommel et al. also reported that other augmentations (Gaussian noise, channel symmetry, channel shuffle, time masking, bandstop filtering) did not provide consistent and significant improvements in performance, which is why we did not include them in our work. The two augmentations SignFlip and TimeReverse showed some improvements in limited settings, but were not included in our study because they do not preserve the structure of EEG data. Finally, while ChannelDropout seemed to provide limited value in the BCI task, a closer look revealed that the dropout rate was tuned to 1 (i.e. dropout of all channels), hinting at an issue with the evaluation task or the training, rather than genuine augmentation benefit.

We will rephrase the relevant part of the related work section in the camera-ready version of the manuscript to make the motivation for our choice of augmentation methods more clear.

Taken together, and given the more comprehensive evaluation of a wide range of augmentations in prior work that informed our study, we believe the three selected augmentations provide an informative picture of contemporary EEG data augmentations. We would also like to add that we do not exclude the possibility that more effective augmentations could be developed as discussed and encouraged in the discussion section of our manuscript.

[1] Aurlien, H et al. ‘EEG Background Activity Described by a Large Computerized Database’. Clinical Neurophysiology, 2004.

[2] Chiang, A. K. I. et al. ‘Age Trends and Sex Differences of Alpha Rhythms Including Split Alpha Peaks’. Clinical Neurophysiology, 2011.

[3] Cellier, Dillan et al. ‘The Development of Theta and Alpha Neural Oscillations from Ages 3 to 24 Years’. Developmental Cognitive Neuroscience, 2021.

Thank you for your helpful review and comments. We appreciate that you found our experiments thorough and methodologically sound.

Limited new insights due to intuitive conclusions

First, we would like to emphasise that our results provide quantitative, more nuanced, surprising, and non-trivial insights.

• We evaluated EEG data augmentations on much larger datasets and across a more diverse set of models and tasks than prior work. Our results show that data augmentations largely failed to improve performance—a conclusion that stands in contrast to their reported success in smaller-scale studies in the literature.
• Our experiments provide quantitative evaluations of the scaling behaviour of different models and the impact of participant diversity across multiple tasks. We would argue that it was not expected that even a more than 50-fold increase in sample size would fail to improve performance in the absence of sufficient participant diversity on TUAB. This result demonstrates how severely model performance can be bottlenecked by the participant count.
• It is currently not common practice to inform data collection with pilot studies that evaluate the importance of participant diversity. As mentioned in the discussion section, our results motivate such pilot studies given the severely participant-count-dominated scaling behaviour on TUAB and CAUEEG on the one hand, and the PhysioNet results on the other hand, where participant diversity was less critical. We are not aware of prior work that provides such actionable guidance using empirical scaling results.

Finally, while some of our results might be well aligned with the intuition of experts in the field, we argue that it is important to test intuitions and substantiate them with empirical results. When two researchers have conflicting intuitions about the existence or importance of participant distribution shifts, our work provides empirical evidence that can inform the discussion.

Role of the analysis in Section 3

In Section 3, we lay out the theoretical foundation that motivates our experiments and provides a framework in which the results can be interpreted.

Section 3 currently concludes with the following summary:

“This analysis tells us that in the presence of a substantial distribution shift, the extent to which one should expect to overfit is governed primarily by the number of participants, rather than the amount of data available per participant, unless one is able to design a model architecture with the appropriate invariances.”

To make the connection to our research questions clear, we will extend this paragraph with the following explanation:

“Under this framework, we address the four research questions outlined in the introduction. We assess the relevance and magnitude of participant distribution shifts through the scaling behaviour of model performance (Q1 & Q2) and the effectiveness of augmentations (Q3) and self-supervised pre-training (Q4) to improve performance by endowing models with useful invariances or boosting effective participant counts.”

Dear reviewer 77HZ,

Please note that this manuscript is not the first to explore how the performance of EEG-based machine learning models is influenced by number of subjects and number of trials. See Banville et. al (2025) https://arxiv.org/abs/2501.15322

Thank you for your efforts in the rebuttal. I appreciate the time and thought you put into addressing the comments.

Thank you for your constructive review and insights. We appreciate your recognition of the generalisability of our findings and the value of directly comparing the effects of data augmentations and self-supervised pre-training.

Limitations section in the appendix

The camera ready copy allows for an additional page in the main paper, in which we will include this limitations section.

Details on train/val/test splits

The official train-test splits were used for TUAB and CAUEEG, and in both cases part of the train split was set aside for validation. For PhysioNet, where labels for the official test split are not publicly available, we created an age and sex stratified split into train, val, and test on the participant level (i.e. without overlapping participants). We will clarify this in the manuscript. For all datasets, the exact splits we used can be reproduced with the shared code.

Discussion of AmplitudeScaling and FrequencyShift augmentations

We will add the following discussion to address why AmplitudeScaling and FrequencyShift may not have improved model performance:

A likely explanation for why AmplitudeScaling and FrequencyShift did not yield consistent accuracy improvements is that these augmentations are not truly label preserving. For example, overall amplitude and differences in amplitude between EEG channels (e.g. asymmetries between corresponding channels on the left and right hemisphere) can provide clinically relevant information [1]. Similarly, frequency shifts can be clinically relevant (e.g. slowing may reflect abnormalities [1] and has been linked to MCI and AD [2, 3]). Alternatively, it is possible that even the smallest amount of training data already provided sufficient diversity of amplitudes and frequency shifts for the model to learn sufficient invariances to these aspects of the data, rendering the augmentations redundant.

[1] Rubin, Devon I. et al. ‘Clinical Neurophysiology’. Contemporary Neurology Ser., 2021.

[2] Cassani, R. et al. ‘Systematic Review on Resting-State EEG for Alzheimer’s Disease Diagnosis and Progression Assessment’. Disease Markers, 2018.

[3] Modir, Aslan et al. ‘A Systematic Review and Methodological Analysis of EEG-Based Biomarkers of Alzheimer’s Disease’. Measurement, 2023.

Discussion of potential augmentations to mitigate participant differences

What augmentations would effectively simulate participant differences is an open research question for which we encourage future work in our discussion section. An important consideration will be the nature of participant distribution shifts as we outline in the Future Work section: for example, if we hypothesise that anatomical differences are the main driver of participant diversity, future work could attempt to develop augmentations to imitate anatomical differences such as skull thickness or head circumference. If participant distribution shifts are instead explained by variability in disease phenotypes, domain knowledge of the specific disease could inform the design of augmentations for it.

Thank you for your thoughtful review and feedback. We appreciate your acknowledgement of the relevance of our research questions and the practical value of our work for guiding data collection.

Uniform subsampling of segments

We confirm that segments were sampled uniformly in our experiments and we agree that contiguous sampling would more accurately reflect data collection in practice. Indeed, random subsampling of segments could lead to an artificially increased segment diversity and overestimation of model performance.

To address this issue, we conducted additional experiments where segments are subsampled contiguously. Specifically, we repeated the experiments for the leftmost heatmap in Figure 3 (varying the number of participants and segments for normality prediction on TUAB with the TCN model) with segments sampled contiguously. We repeated these experiments for 5 different seeds. The results show that our conclusions still hold. In particular, even when segments are sampled contiguously instead of uniformly, collecting additional segments is not a reliable way to compensate for a lack of participants.

We will include the results of this control experiment in the manuscript, showing accuracy heatmaps (with the standard error of the mean across seeds) for uniformly sampled vs contiguously sampled segments side-by-side. Since we are unable to share the plots via openreview, we provide the results in the form of a table here.

Uniform sampling (average accuracies along with the standard error of the mean across 5 seeds):

Contiguous sampling (average accuracies along with the standard error of the mean across 5 seeds):

Too much content

We have considered splitting the research questions into separate pieces of work. However, we believe that it is valuable to present them together under the theoretical framework introduced in Section 3. By including research questions 3 and 4, investigating the effectiveness of common machine learning methods to address data scarcity, we are able to present a more complete picture of current data limitations in EEG-based machine learning.

No novel methods

The NeurIPS2025 Reviewer Guidelines state that: “[...] originality does not necessarily require introducing an entirely new method. Rather, a work that provides novel insights by evaluating existing methods, or demonstrates improved efficiency, fairness, etc. is also equally valuable.”

We believe it is fair to claim this for our work.

Thank you for your review and constructive feedback. We are glad you valued our systematic empirical analysis.

Limited number of tasks, models, and datasets

We aimed to design a diverse and sufficiently broad set of conditions to support meaningful and generalisable conclusions. Our experimental setup provides a solid basis to answer our research questions, and experiments on additional datasets would not change our conclusions.

• We deliberately included three models with fundamentally different architectures: TCN (CNN-based), mAtt (geometric deep learning using covariance representations), and LaBraM (Transformer-based). These span a range of methodological paradigms explored in recent EEG literature and reduce the likelihood of model-specific conclusions.
• We focused on large-scale datasets (1000+ participants), which are rare in EEG research [1]. Large datasets are essential for our experimental design, as they allow us to systematically vary the number of participants and segments over several orders of magnitude. Our work could not have been done with a large number of small datasets. Additionally, evaluations on small datasets, while easier to scale, are often noisy and less reliable.
• Compared to prior work, we are not aware of any EEG-based ML study with evaluations on a higher number of large (1000+ participants) datasets, diverse model architectures, and different tasks.
• Additional datasets or tasks would not change the answers to our research questions. Normality prediction on TUAB and dementia diagnosis on CAUEEG already provide striking examples of severe participant distribution shifts, whereas sleep staging experiments on PhysioNet showed that there are specific applications where it is possible to achieve good performance even with limited participant counts. So while additional tasks and datasets could have some value to inform application-specific future work, we do not require more experiments to answer the research questions in this paper.

Finally, we would like to emphasise the computational scale of our current experiments. For example, for the baseline experiments on TUAB alone, we evaluated 49 different data regimes (combinations of participant and segment counts) for each of the three models and repeated each configuration with 25 random seeds, resulting in 3675 training runs.

[1] Roy, Yannick et al. ‘Deep Learning-Based Electroencephalography Analysis: A Systematic Review’. Journal of Neural Engineering, 2019.

Missing conclusion section

We aimed to use the Discussion section as an extended Conclusion section. We will include a short summary of the main points of this Discussion section in a new Conclusion section for the camera ready version.

More discussion of underlying causes behind observations

• Observations for Q1 and Q2 are addressed through the analysis in Section 3, which describes a theoretical framework under which the importance of participant diversity can be explained. We link back to this analysis in the results section, e.g. line 264.
• Observations for Q3 are discussed in Section 4.3. For instance, we discuss underlying reasons why the PhaseRandomization augmentation failed to improve performance in line 291. We will extend the camera-ready version of the manuscript with further discussion about AmplitudeScaling and FrequencyShift (see below).
• Observations for Q4 are discussed in Section 4.4. We will extend the camera-ready version of the manuscript with further discussion (see below).
• In addition, model differences are discussed in a dedicated section in A.5 with references and additional discussion in the results section, e.g. lines 248 and 287.

Extended discussion of observations about AmplitudeScaling and FrequencyShift:

A likely explanation for why AmplitudeScaling and FrequencyShift did not yield consistent accuracy improvements is that these augmentations are not truly label preserving. For example, overall amplitude and differences in amplitude between EEG channels (e.g. asymmetries between corresponding channels on the left and right hemisphere) can provide clinically relevant information [1]. Similarly, frequency shifts can be clinically relevant (e.g. slowing may reflect abnormalities [1] and has been linked to MCI and AD [2, 3]). Alternatively, it is possible that even the smallest amount of training data already provided sufficient diversity of amplitudes and frequency shifts for the model to learn sufficient invariances to these aspects of the data, rendering the augmentations redundant.

[1] Rubin, Devon I. et al. ‘Clinical Neurophysiology’. Contemporary Neurology Ser., 2021.

[2] Cassani, R. et al. ‘Systematic Review on Resting-State EEG for Alzheimer’s Disease Diagnosis and Progression Assessment’. Disease Markers, 2018.

[3] Modir, Aslan et al. ‘A Systematic Review and Methodological Analysis of EEG-Based Biomarkers of Alzheimer’s Disease’. Measurement, 2023.

Extended discussion of observations for self-supervised pre-training:

Pre-training exposed the model to a larger diversity of EEG data from additional participants than those seen during fine-tuning. The positive results suggest that, even without labels, this has allowed the model to learn useful representations that transfer to diverse downstream tasks.

If there are other observations that the reviewer believes would benefit from further discussion, could they let us know more specifically what they are?

Motivation for selected models

We selected three influential and widely used models with distinct architectures to obtain a diverse perspective on EEG model behavior and reduce the risk of model-specific effects.

• TCN is a convolutional neural network (CNN), which was developed for generic sequence modeling and has since been widely adopted in EEG research. CNNs remain a strong baseline in EEG-based machine learning.
• mAtt is a geometric deep learning model specifically developed for EEG. It encodes signals as sequences of covariance matrices and exemplifies a line of work that incorporates Riemannian geometry to leverage domain-specific structure.
• LaBraM is a transformer architecture. As in other domains, transformers have become a common architecture choice in recent works, particularly for the development of foundation models.

These models provide a good picture of the landscape of the current state-of-the-art in EEG-based machine learning.

Motivation for selected augmentations

As discussed in the related works section, our selection was informed by the systematic evaluation of EEG augmentations conducted by Rommel et al. In their benchmarks on two tasks (sleep staging and BCI motor imagery), PhaseRandomization (termed FTSurrogate) consistently emerged as the most effective augmentation. FrequencyShift showed small improvements when data was more limited. Inclusion of the FrequencyShift augmentation was further motivated, because differences in peak frequencies have been shown to be associated with demographic variables like age and sex [1, 2, 3], making it an interesting augmentation to study in the context of participant diversity. In addition, we also studied AmplitudeScaling, an augmentation introduced by Mohsenvand et al. in the context of a contrastive learning approach, but not evaluated in the review by Rommel et al.

Importantly, Rommel et al. also reported that other augmentations (Gaussian noise, channel symmetry, channel shuffle, time masking, bandstop filtering) did not provide consistent and significant improvements in performance, which is why we did not include them in our work. The two augmentations SignFlip and TimeReverse showed some improvements in limited settings, but were not included in our study because they do not preserve the structure of EEG data. Finally, while ChannelDropout seemed to provide limited value in the BCI task, a closer look revealed that the dropout rate was tuned to 1 (i.e. dropout of all channels), hinting at an issue with the evaluation task or the training, rather than genuine augmentation benefit.

We will rephrase the relevant part of the related work section in the camera-ready version of the manuscript to make the motivation for our choice of augmentation methods more clear.

Taken together, and given the more comprehensive evaluation of a wide range of augmentations in prior work that informed our study, we believe the three selected augmentations provide an informative picture of contemporary EEG data augmentations. We would also like to add that we do not exclude the possibility that more effective augmentations could be developed as discussed and encouraged in the discussion section of our manuscript.

[1] Aurlien, H et al. ‘EEG Background Activity Described by a Large Computerized Database’. Clinical Neurophysiology, 2004.

[2] Chiang, A. K. I. et al. ‘Age Trends and Sex Differences of Alpha Rhythms Including Split Alpha Peaks’. Clinical Neurophysiology, 2011.

[3] Cellier, Dillan et al. ‘The Development of Theta and Alpha Neural Oscillations from Ages 3 to 24 Years’. Developmental Cognitive Neuroscience, 2021.