The VEP Booster: A Closed-Loop AI System for Visual EEG Biomarker Auto-generation

1. Lack of comparison with other methods. First of all, the authors provide comparisons of NN-modified Gabor-like structures and measure their ability to elicit the frequency entrainment, comparing it to the natural images. SSVEP paradigm used by the authors is well known and there have been numerous studies dating 10+ years ago and the recent ones aimed to optimize the spatial and temporal parameters of the visual stimuli used in the SSVEP paradigm in order to improve the detectability. While the authors provide numerous references to other approaches in Section 2 including traditional signal processing techniques, machine learning (and neural network)-based techniques and Spiking Neural Networks, they do not compare their approach with at least some of the most closely related ones.
2. Small dataset. The proposed dataset includes 9 subjects and 4500 trials. This is generally a pretty small amount of data. Therefore, it is not clear how well the approach scales and generalizes across subjects. Also, one of the experiments authors do is to train a neural network to classify if a subject was under SSVEP stimulation or not. They use a traditional 80-20 split which leaves only 900 samples for testing. It is also possible that neural networks’ quality was affected by the lack of data during training. At the same time the modern SSVEP paradigms are capable of designing neurointerfaces with 16-32 command where each command is encoded with a unique visual pattern flickering at a given rate. Pushing the limits of such SSVEP solutions with the proposed approach and comparing it against the earlier methods would significantly increase the impact of the study.
3. Potential model-specific ML issues. Mode collapse. GANs are famously prone to a mode collapse where a model starts to generate a really small variety of outputs. Considering that authors were not only using GAN, but also finetuning it, it is important to address if a GAN actually learned to produce images well and was it affected by a mode collapse. This seems especially important to address since authors talk about convergence of the system to the same latent vectors (as seen in Figure 8 and Figure A.1). As it could be a sign of a mode collapse affecting the closed-loop experiment. Also, are the authors sure that they truly need large NN architectures to solve their task?
4. Reproducibility. It would be very useful if authors could provide more details about how their model was trained: optimizers, loss functions, etc. Also, authors utilize a dataset with 9 people, but in the Introduction (lines 98-99) they say they will open-source data only from 3 subjects. This raises the reproducibility concerns. Especially, considering weakness (1): authors did not try other models on their data and other authors will not be able to train their models for direct comparison with this approach without having access to the same full dataset.
5. Incorrect statements. While I understand that the authors are primarily people working on the ML, I still think the text has to be monitored for incorrect and very naive statements, For example such as “The reason for selecting 4 Hz is that it is the frequency of the brain’s background noise, which aids significantly in evaluating whether the testing system can effectively evoke the target frequency while suppressing the background noise.” - It is well known both neuroscience and EEG signal processing community that the brain’s background noise has PDF of the form P(f) ~ 1/f^alpha and is not centered around 4 Hz. 4 Hz fall in the range of so called theta-band known to be associated with memory related processes.

1. The generated stimuli are overly simple, as the model primarily produces variations in brightness, stripe patterns, and checkerboards. These could likely be constructed with a simpler model with fewer parameters, rather than a complex DCGAN, which would reduce optimization challenges. Additionally, if a complex model like a DCGAN is employed, it would be feasible to apply this framework with natural images, which would be significantly more flexible and practical than the current stimuli.
2. Typos: The graphic in Figure 5a appears stretched； Figure A.2, subtitle for the 3rd row, "CCNN" should be corrected.

On the one hand, the technical contributions are limited, and also only briefly described. Therefore, I fear that this submission will only be of interest to a small group of researchers in the ICLR community. On the other hand, the neuroscientific aspects are also only superficially described. How are the stimuli precisely designed? Why do we need a GAN for this purpose? I believe that the stimuli are quite simple, and they could be rendered by a simple scripts instead of a GAN. What can be learn from the optimized patterns in Fig 4 in terms of biophysics? It is also not clear how this paradigm could be used in practical applications. I recommend providing specific examples of potential practical applications for the proposed system, or to elaborate on the challenges that might need to be addressed for real-world implementation.

In summary, the paper may only have limited appeal to the methods-oriented audience on the one hand and neuroscience community on the other hand, since the methodology and interpretation of the results is too superficial.

1. There are multiple unsubstantiated claims about EEG: the intro talks about “intrinsic preferences of specific neurons” however there is no way to identify those with EEG. This also comes up at line 211 (“maximal response within V1”), line 241 (“4 Hz [...] is the frequency of the brain’s background noise”) and line 269 (“generating images that preferentially activate biological visual cortex V1”).
2. A lot of technical details require clarification including the way latent vectors are generated, modified and sampled to create trial images (Q1-3), and how the two main metrics used in the study are computed (Q4). Results in Figure 3 are not clearly labeled/explained.
3. While I agree that the data that was collected in this study would be useful to share, listing it as one of the three core contributions of the paper is disputable as it has not been shared yet.
4. The text needs to be proofread.