Joint Training Does Not Transfer Information between EEG and Image Classifiers

In order to confute Palazzo et al.’s work, the authors apply PCA to show that most of the variance of image and EEG embeddings is explained by the first 40 principal components. Based on my understanding of the work, I do not agree with the conclusions drawn by the authors. The fact that the first 40 PCs are able to describe image embeddings is expected, since the image encoder in Palazzo et al. is pre-trained on ImageNet, which includes the 40 classes used in the dataset: it is reasonable that features extracted by the encoder linearly separate the target classes. The fact that the first 40 PCs are able to describe EEG embeddings sounds to me as a confirmation that the EEG encoder effectively extracts neural activity patterns that correlate well with image content. The authors imply that this correlation is a consequence of the “confounded” nature of the EEG dataset: however, this statement seems to be subject of debate (see Li et al., 2021 and Ahmed et al., 2022, from the paper; and [A] below). It should also be noted that the authors claim that “Several independent lines of research have completely refuted a large body of completely flawed work”, although — based on the references — these “independent lines of research” exhibit a significant coauthorship overlap.

From these considerations, I do not believe that the paper makes a strong point in proving its case. Additionally, besides my concerns on the soundness of the work, in my opinion the paper is not suitable for ICLR, since it does not provide any methodological contribution, and I do not envisage a lot of interest but for a very specific niche of the audience. I think this kind of work is more suitable as a comment paper to the relevant prior publication.

[A] Palazzo, S., Spampinato, C., Schmidt, J., Kavasidis, I., Giordano, D., & Shah, M. (2020). Correct block-design experiments mitigate temporal correlation bias in EEG classification. arXiv preprint arXiv:2012.03849.

Clarity of Presentation

The paper's presentation is not optimal. While its aim is to refute all points made in (Palazzo et al., 2021), it does not systematically or clearly introduce the content from (Palazzo et al., 2021). A rigorous critique requires relevant equations and plots for clear justification or intuitive understanding, both of which are absent in the paper. Tables 2 and 3, in particular, are challenging to decipher.

Audience Suitability and Depth of Analysis

The content might not be best suited for the ICLR audience. While the act of rebuttal is acceptable, the paper falls short in offering meaningful insights, explanations, or suggested improvements. It lacks adequate context, making it burdensome for a reviewer to consult another paper for making a fair judgment on this one. Claims such as "the dataset and data collection methods are fundamentally flawed" lack the requisite evidence to be convincing. Moreover, multiple works (e.g., Bai et al., 2023; Zeng et al., 2023; and others) are critiqued without adequate explanations or justifications.

The basic question needs a mathematical/statistical formulation that can guide experimental design and give context for the conclusion. The experimental design needs better justification - the "why" for each step. Missing uncertainty estimates. Please clarify to allow the reader to weigh the evidence.

The related work of the manuscript could be expanded, by discussing more recent works and datasets in the area of EEG-based object recognition (e.g. Gifford et al. (2022)) References: Gifford AT, Dwivedi K, Roig G, Cichy RM. A large and rich EEG dataset for modeling human visual object recognition. NeuroImage. 2022 Dec 1;264:119754.

Dataset descriptions are not sufficient, hence the manuscript is not self-contained. Details which may be found by reading the relevant references, are missing (e.g. why are data only from subject 6 picked?). Such details should be described at some level within the manuscript. Moreover, I believe that the names which are used to denote the datasets throughout the manuscript ("theirs", "block", "randomized") could be changed so as to become more descriptive. E.g. the name "theirs" does not help the reader to understand where does it refer to. Also the name "block" can be confusing to some readers, as actually both (Palazzo et al. (2021)) and (Ahmed et al. (2021)) datasets have a block design. EEG signal preprocessing details are not mentioned, which also renders the manuscript less self-contained.

Section 2 (Method) could be organized in a more clear way. At its current form, it is difficult for a reader to follow the flow of this section. Some low level details (e.g. numbers related to the PCA analysis, numbers of classes in ImageNet and the EEG datasets, which cases count as misclassifications etc.) are presented in this Section, making it difficult to obtain a high level understanding of each experiment, e.g. the motivation behind it, an abstract experimental setup, the expected conclusions etc.

Section 3 (Results) needs to be presented in a more organized and coherent way. At its current form, this section does not enable a gradual understanding of the experimental analysis, as it simply describes the result of each Table, without allowing for a more gentle introduction to each experiment.

There are several presentation issues on the Tables of the manuscript. For example, regardings Tables 1, 2 and 3, instead of matching row IDs to particular image encoding models and particular datasets, it would be better to directly mention these information inside the Tables (instead of providing instructions on how to interpret these information through the caption). There are two errors in the caption of Table 2 (error 1: "Rows i-vii 'their'" should actually be "Rows i-viii 'their'", error 2: "Rows v-vii" should actually be "Rows v-viii").

Despite the interesting takeaways of this work, I find the value that it would bring to the community of ICLR to be below the standards. Several issues related to the method of Palazzo et al. (2021) have been discussed in previous works (e.g. Ahmed et al. (2021) and Li et al. (2021)). Raising additional issues about the method of Palazzo et al. (2021) contributes to an already opened discussion, yet it does not positively show any way forward. The authors state that "We make no claim that it is difficult or impossible to perform the central task attempted by Palazzo et al. (2021)", however the line of research that the manuscript follows, and further explores, would highly benefit from additional methodological contributions that could yield improvements on the studied task.

There are some typos, grammar, syntax and other issues throughout the manuscript, e.g.:

• " 'their,' " (caption of Table 1)
• "and!xxi-xxiv" (caption of Table 3)
• The authors should replace the word "falsified" (page 9), with a more proper one.

References: Ahmed et al., "Object classification from randomized EEG trials", IEEE CVPR 2021 Li et al., "The perils and pitfalls of block design for EEG classification experiments", IEEE TPAMI 2021

1. The focused nature of the manuscript, primarily critiquing a specific journal paper, raises questions about its suitability for a broader conference audience.

2. The manuscript's contribution to advancing general EEG decoding techniques remains ambiguous.

3. The manuscript's novelty is somewhat challenging to ascertain, as it primarily focuses on validating the work of others. The authors might consider adding a general discussion section that elaborates on any novel insights or methodologies that their validation efforts have yielded, thereby contributing to the field's body of knowledge.