Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

Thank you for your careful review and comments. Below please find our point-to-point responses to your comments.

Q1. “The pipeline is already well-established in the field.”

Please kindly see the relevant answer in Global rebuttal: Q1 and Q2.

Q2. How the VAE is supposed to provide low-level image features.

Thanks to the reviewer for pointing out this part, and we may not have expressed it clearly in the manuscript. Due to limited space, please refer to the reply to reviewer (zVcP).

Q3. “How can we get clear reconstruction results when only 50 ms? ”

Please kindly see the relevant answer in Global rebuttal: Q3.

Q4. Should we think the retrieval was based on some low-level information?

We consider the high-level visual features should play a major role in the visual retrieval task.

On the one hand, there is much noise in the scalp EEG, and the dynamic activities of the temporal cotex may not be accurately captured by the EEG, while the occipital cortex continues to have a strong response.

Furthermore, when we perform the retrieval task, we use the high-level visual features after the alignment of CLIP and EEG. So the semantic-level visual features should play a major role in the visual retrieval task.

Q5. The visual response is very quick after the onset, finished before 200 ms....

Please also kindly see the relevant answer in Global rebuttal: Q3.

Q6. On page 9 Line 251, It can’t be concluded that...

In THINGS-MEG, each image in the THINGS-MEG was displayed for 500 ms. There was a fixed time for each image of 1000 ± 200 ms.

However, in THINGS-EEG, each of image was presented for 100 ms. For preprosessing and training, we segmented the EEG data from 0 to 1000 ms after the stimulus onset into trials. We speculate that our excellent performance in the EEG visual decoding task is due to the larger data size of the THINGS-EEG (the more trials, the better the decoding performance). That is, in the visual decoding task, scaling low still works.

Due to the different experimental paradigms of THINGS-EEG and THINGS-MEG and the characteristics of the two types of data, we only report the phenomena we observed. We hope that these results can trigger further discussion and promote the development of the community.

Q7. It would be more rigorous to test the model only once after it is fully trained.

As described in the manuscript, what we used in Figure 5c and Figure 5d is the result of the highest test accuracy of all methods in the process of training. In Table 7 in the Appendix G, we present the results after 10 tests and take the average results. Regardless of the test strategy, our method performs far ahead of other methods.

Q8 .Which input to the diffusion process is most critical for image generation ?

This is also an interesting question. In fact, after our tests, the EEG embedding aligned with CLIP is the most necessary. The output after VAE reconstruction also comes from EEG and the reconstructive image is very blurry, so it contributes very little. On the other hand, the text obtained from the generated caption is very similar to the image embedding, so it also contributes very little.

Q9. How does the model ensure that it captures low-level features?

Please kindly find the related explanations in reviewer (zVcP) in Q2. The low-level and high-level here refer to the representation that obtained from the pixel-level or the semantic-level after CLIP alignment with [1][2]. So it is not the low-level and high-level areas of visual cotex in neuroscience.

Q10. What is the impact of the pre-trained models on the final performance?

During the experiment, we found that the pre-trained language model seemed to have little impact on the final results, whether in the representation learning stage or the generative model stage. This may be because CLIP itself is a pre-trained model after the image and text are aligned. So the contribution of the text is not so important.

On the other hand, the generative model only reconstructs the semantically correct category. Therefore, it does not improve the decoding accuracy, but only guarantees the quality of the reconstructed image.

Q11. Provide more details about the cross-subject settings, and do they enhance cross-subject capability?

Please find the explanations in reviewer (zVcP) in Q7. The objective is to retain both subject-independent joint EEG representations and subject-specific independent EEG representations during training.

Q12. Could you comment on this observation in Figure 11a ?

As explained in Q10, this may be due to the fact that CLIP is a pre-trained model with aligned image and text features. We consider that these works [3][4] that use CLIP to align neural signals will also have this property.

Reference

[1]Scotti P, Banerjee A, Goode J, et al. Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors[J]. Advances in Neural Information Processing Systems, 2024, 36.

[2]Takagi Y, Nishimoto S. High-resolution image reconstruction with latent diffusion models from human brain activity[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 14453-14463

[3]Du C, Fu K, Li J, et al. Decoding visual neural representations by multimodal learning of brain-visual-linguistic features[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(9): 10760-10777.

[4]Zhou Q, Du C, Wang S, et al. CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding[C]//The Twelfth International Conference on Learning Representations.

[5]Benchetrit Y, Banville H, King J R. Brain decoding: toward real-time reconstruction of visual perception[C]//The Twelfth International Conference on Learning Representations.

[6]Song Y, Liu B, Li X, et al. Decoding Natural Images from EEG for Object Recognition[C]//The Twelfth International Conference on Learning Representations.

We appreciate the reviewer's thoughtful feedback and recognize the significance of the highlighted concerns.

(1) Thank you for your suggestion. We will add the results on the validation set in the official version of the paper. In fact, we initially considered that if we adopt the same approach as in [1], that is, to divide a small number of trials in the training set as the validation set for evaluating the model, this may violate the zero-shot task setting. Different from those classification and retrieval tasks with known categories, due to our unique processing form (in a dataset with a total of 1864 categories, we use 1654 categories of samples as the training set and the remaining 200 categories as the test set), so we hope to always evaluate the performance of the model in a zero-shot form.

Our original approach was to train the models of different encoders for a sufficient number of rounds (30 epochs in the experiment) to ensure model convergence, and test them on the test set in the last 10 epochs. Since all models use the same evaluation method, and we strictly control the random seed and ensure that the data is not leaked, the final evaluation results are also unbiased.

(2) This is another interesting question. First of all, our statement has some typo. What we want to express is "The low-level and high-level here refer to the representation that obtained from the pixel-level after VAE alignment with or the semantic-level after CLIP alignment with". Past work [2] has found that in the denoising stage of the early diffusion model, z signals (corresponding to the VAE latent in our framework) dominated prediction of fMRI signals. And during the middle step of the denoising process, zc predicted activity within higher visual cortex much better than z. However, please note that this is only an analysis based on decoding accuracy. These analyses do not have a strong neuroscience causal relationship. We still cannot conclude that the low-level features of fMRI are modeled by VAE.

From our experimental results, the more VAE latent is used in the denoising process, the more certain the overall reconstructed image is and the less details it has. Conversely, the more details it has. Therefore, the contribution of VAE latent and clip latent to reconstruction tends to be a balance relationship. Our future work should focus on achieving more brain-like decoding, paying attention to both well low-level and high-level reconstruction, rather than maintaining an either-or relationship between the two.

Reference

[1] Song Y, Liu B, Li X, et al. Decoding Natural Images from EEG for Object Recognition[C]//The Twelfth International Conference on Learning Representations.

[2] Takagi Y, Nishimoto S. High-resolution image reconstruction with latent diffusion models from human brain activity[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 14453-14463

Thanks for your kind reply. I still have concerns: 1) Q7, Why not use a validation set? It's not convincing to use test sets in training processing. 2) Q9, what are the gains of low-level features from VAE for the overall framework, if they are pixel-level representations after CLIP alignment, instead of low-level features defined in vision?

Sorry to bother you. We are about to run out of time to respond.

We have made complements and explanations for this work with the help of all the reviews. We would be grateful if you could confirm whether the rebuttal meets your expectations and if there is any other suggestion.

Thank you once again for your time and insightful comments!

Thank you for your careful review and comments. Below please find our point-to-point responses to your comments.

Q1. “First, its motivation is based on the signal differences between EEG and fMRI, concluding that EEG's performance limitations are due to constraints in decoding and reconstruction frameworks......”

Please kindly see the relevant answer in Global rebuttal: Q1 and Q2. This paper has two main motivations:

First, existing fMRI-based visual decoding and image reconstruction methods [1][2] are very advanced, but they are limited by the temporal resolution, portability and cost of fMRI, so these technologies cannot be applied in practice.

Second, the idea of the NICE [3] and B.D. [2]’s method are simple but very general. Previous work on visual decoding using fMRI often used ridge regression as the encoder. However, using EEG is different from fMRI, as it is full of noise and the original signal-to-noise ratio is low. It requires a sufficiently strong prior and a more complex model structure to extract the feature maps. We carefully studied the shortcomings of different approaches in EEG decoding [5][6]. Inspired by NICE [3], and based on the theory of treating channels as patches in [4], we introduced joint subject training, which far aheads of NICE, and gained the ability to adapt to new subjects like [7][8].

The contribution of this paper is not to improve decoding performance too much only by stacking tricks. The main purpose is to provide evidence for the neuroscience and machine learning communities: using our framework, even compared with fMRI, EEG can provide competitive performance.

Q2. “The generation of images from 50ms EEG signals, as shown in Figure 7, contradicts established neuroscience principles. This suggests that the visual stimulus reconstruction framework heavily relies on the image generation model, which may have limited significance for the field of neuroscience.”

Please kindly see the relevant answer in Global rebuttal: Q3. We may not have explained it clearly in the article, but there are similar expressions in [2]. According to our results in Appendix H.1 Accuracy for time windows, Figure 29 shows that for all embeddings, a clear peak can be observed for windows ending around 200-250 ms after image onset. Comparing Figure 28 and Figure 29, we can see that, unlike [2], our time window results on THINGS-EEG dataset do not have a second peak, which may be mainly affected by the experimental paradigm.

Please pay attention to the caption in Figure 7 and the reconstruction results from 0 to 50 ms and 0 to 250 ms. It can be seen that the image reconstructed at 50 ms is completely messy and has no semantics; the semantics of the reconstructed image after 250 ms is basically correct and gradually stabilizes. Similar results are also shown in Figure 4 A of [2]: their method can also reconstruct high-quality images at 100ms. This just shows that our reconstruction results are in line with the neuroscience prior.

In addition, we provide three different tasks including image classification, image retrieval and image reconstruction. The excellent performance in image classification and image retrieval tasks shows that the decoding of visual stimuli is reasonable. Since the submission is to NeurIPS, the top machine learning conference, it is necessary for us to use the latest technology to ensure the quality of image reconstruction on this basis, which reflects the latest technological progress.

Reference

[1]Scotti P S, Tripathy M, Torrico C, et al. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data[C]//Forty-first International Conference on Machine Learning.

[2]Benchetrit Y, Banville H, King J R. Brain decoding: toward real-time reconstruction of visual perception[C]//The Twelfth International Conference on Learning Representations.

[3]Song Y, Liu B, Li X, et al. Decoding Natural Images from EEG for Object Recognition[C]//The Twelfth International Conference on Learning Representations.

[4]Liu Y, Hu T, Zhang H, et al. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting[C]//The Twelfth International Conference on Learning Representations.

[5]Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces[J]. Journal of neural engineering, 2018, 15(5): 056013.

[6]Song Y, Zheng Q, Liu B, et al. EEG conformer: Convolutional transformer for EEG decoding and visualization[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 31: 710-719.

[7]Xia W, de Charette R, Öztireli C, et al. UMBRAE: Unified Multimodal Decoding of Brain Signals[J]. arXiv e-prints, 2024: arXiv: 2404.07202.

[8]Zhou Q, Du C, Wang S, et al. CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding[C]//The Twelfth International Conference on Learning Representations.

Thank you for your careful review and comments. Below please find our point-to-point responses to your comments.

Q1. “I think it hasn’t been clarified in the main text whether....”

Regarding 3.2 to 3.5: We present the bar graphs of performance in the Section 3.2 EEG Decoding Performance, which are the absolute results of the within-subject. The detailed tables of retrieval performance are in Appendix H.1. In the Section 3.3 Image Generation Performance, we used the EEG data of subject 8 as a representative for image reconstruction and calculated mertics.

There are statistical images reconstructed by a single subject in Appendix G. The experiments related to ‘Section 3.4 Temporal Analysis’ and ‘Section 3.5 Spatial Analysis’ were all conducted on the data of subject 8. We will follow your suggestion and add clear sentences in the titles of the figures and tables for explanation.

Q2. What model is inside the frozen “VAE image encoder” in Figure 2? 

The details of this part are similar to [1]. We now make complementary explanations: The VAE model from the pre-trained StableDiffusion-v2.1-base is used for low-level image reconstruction, as shown in Figure 2. We use the latent obtained by VAE to align the latent of EEG and then reconstruct the low-level image from EEG. First, the preprosessed EEG passes through an MLP and upsampling CNN to obtain a 4x32x32 latent, and then passes through MSE loss, contrastive loss, reconstruction loss, and aligns the latent from the original image 3x256x256 after VAE encoding. We pass EEG latent through the VAE decoder to get a blurred 3x256x256 image, and then input the image into SDXL-turbo as low level guidance.

Q3. The authors didn’t introduce the “image2text” component (BLIP2).

Thank you for pointing out a typo, and we didn't make it clear here. The BLIP2 model [7] is indeed used in this work, but it is only used to extract text features from the original image for EEG classification. In the image reconstruction stage, to provide guidance from text features, we adopted the same approach as [5], using the GIT model [6] to obtain text descriptions from image embeddings in the reconstruction stage, which is then input into the SDXL-turbo decoder as a condition.

Q4. Why are these two papers not compared? [2][3]

In [2] and [3], both were conducted on a controversial dataset [4]. The experimental paradigm of this dataset was once accused of having a block design error, resulting in the contribution of decoding not entirely coming from the category itself. The experimental paradigm of the THINGS-EEG does not have this problem, because the training set of this dataset has 1654 categories and the test set has 200 categories, and the categories of its training set and test set are completely different.

Q5. Regarding the ‘PixCorr’ metric:

Due to the limited length of the table and the fact that the low-level metric already has SSIM, this is omitted in the manuscript. The performance on different benchmarks is summarized as follows:

Q6. For Figure 4, the plots are impressive, but ....

Thank you for your suggestion, for Figure 4, our original intention was to make the radar chart as simple and intuitive as possible. Detailed performance tables for decoding and reconstruction are given in Appendix H.1 Table 7 and in the rebuttal to the reviewer (asin) for more detailed results.

Q7. The functioning of the shared token and subject tokens.

Similar to [8], we leverage joint subject training to adapt to new subjects. Once a model is trained, it can be used for both reasoning about known subjects (subject-specific tokens) and reasoning about unknown subjects (shared tokens).

Reference

[1]Scotti P, Banerjee A, Goode J, et al. Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors[J]. Advances in Neural Information Processing Systems, 2024, 36.

[2]Bai, Yunpeng, et al. "Dreamdiffusion: Generating high-quality images from brain eeg signals." arXiv preprint arXiv:2306.16934  (2023).

[3]Lan, Yu-Ting, et al. "Seeing through the brain: image reconstruction of visual perception from human brain signals." arXiv preprint arXiv:2308.02510 (2023).

[4]Spampinato, C.; Palazzo, S.; Kavasidis, I.; Giordano, D.; Souly, N.; and Shah, M. 2017. Deep learning human mind for automated visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6809–6817.

[5]Ferrante M, Boccato T, Ozcelik F, et al. Multimodal decoding of human brain activity into images and text[C]//UniReps: the First Workshop on Unifying Representations in Neural Models. 2023.

[6]Wang J, Yang Z, Hu X, et al. GIT: A Generative Image-to-text Transformer for Vision and Language[J]. Transactions on Machine Learning Research.

[7]Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//International conference on machine learning. PMLR, 2023: 19730-19742.

[8]Zhou Q, Du C, Wang S, et al. CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding[C]//The Twelfth International Conference on Learning Representations.

[9]Benchetrit Y, Banville H, King J R. Brain decoding: toward real-time reconstruction of visual perception[C]//The Twelfth International Conference on Learning Representations.

[10]Fatma Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using generative latent 342 diffusion. Scientific Reports, 13(1):15666, 2023.

Thank you to the reviewer for the impressive questions and suggestions.

We are very grateful to the reviewer for pointing out the technical details that were not clarified in our article. The content presented in this article is quite comprehensive and sufficient, but we sincerely hope the reviewer to pay further attention to our innovation and unique contributions:

1. Technically, we consider the positional relationship between channels and the modeling of the time dimension, so we introduce a channel-wise patch embedding and feedforward neural network in the time dimension (Figure 3). EEG data is different from fMRI, which requires sufficiently effective feature extraction and neuroscience-specific inductive paranoia to maximize the performance expectations of decoding - we have achieved state-of-the-art on different tasks (Figure 4). Finally, in the uploaded pdf file, we provide comprehensive evaluation results.

2. In terms of scientific insights, based on strong performance, we further provide analysis results of different dimensions such as time distribution (Figure 7), spatial distribution (Figure 8), representation distribution (Figure 11) and concept distribution (Figure 12), which strongly proves the effectiveness, scalability, interpretability and causality of our framework for EEG decoding and reconstruction.

3. In terms of potential impact, our work enhances the interpretability of the model on existing neural decoding tasks as much as possible, including decoding of cognitive concept foundations (Figure 12) and revealing neural mechanisms (Figure 8). Furthermore, we focus on the study of causal relationships. In order to understand the reasons for the high decoding accuracy, we conducted a number of spatial-temporal ablation experiments, compared and demonstrated in detail the corresponding viewpoints in EEG decoding and neuroscience. In addition, in order to prove the effectiveness of the results, future work will focus on the verification of real-world applications - we are collecting enough EEG data from different subjects for full parameter or efficient parameter fine tuning to further verify the effectiveness of our framework.

4. We have updated codes in the anonymous code link (https://anonymous.4open.science/r/Visual_Reconstruction-AC56), and all codes provided in the review area will be refined and be public after the anonymous phase. And we provided examples of reconstructing images with different random seeds for growing windows across time scales (README.md). From the two example stimulus images provided, the reconstructed images show uncertainty between 0 to 50 ms and 0 to 250 ms because the stimulus has just arrived at the primary visual cortex and it takes time for the brain to react and be captured, the reconstructed image during this period shows uncertainty (probably due to noise unrelated to the stimulus). Here, since the prior of natural images is added to the two-stage framework, high quality images can be reconstructed even from 0 to 50 ms and 0 to 250 ms. Then, with the accumulation of visual stimulus information, the semantics of the reconstructed images gradually become clear. After 500 ms, the images decoded by EEG tend to be stable, which means that there may be no new information added.

Thank you for your time and thorough comments! Below please find our point-to-point responses to your comments.

Q1. How did you align the channel heterogeneity? Is it a zero-shot approach?

Regarding Section 3.1: ”To verify the versatility of ATM for embedding electrophysiological data, we tested it on MEG data modality using the THINGS-MEG dataset" :

What we mean in the text is that in order to prove that our framework is universal for general embedding electrophysiological data, we also test the performance of our framework on the THINGS-MEG dataset. As can be seen in Figure 4, the MEG dataset and the EEG dataset are different evaluation systems because they are trained and evaluated separately. Therefore, the issue of channel heterogeneity is not involved here. Similar to the THINGS-EEG dataset, we also performed zero-shot testing on the THINGS-MEG dataset.

Q2. “It would be helpful to clarify and explain the experimental settings of these additional datasets. ”

Regarding Table1: In order to intuitively demonstrate the advantages of our framework, we compared the performance of our method with different methods on EEG, MEG and even fMRI dataset. Under the zero-shot setting, our framework makes the image reconstruction performance surpass the performance of [1] on MEG data, and is closer to the performance of current methods [1][2][3] on fMRI data, which highlights the superiority of our framework and makes image decoding and reconstruction more practical.

Q3. “It would be better to demonstrate the effectiveness of the proposed framework compared to existing fMRI-image decoding methodologies. ”

Please kindly see the relevant answer in Global rebuttal: Q1 and Q2. We believe that the differences in the characteristics of fMRI and EEG themselves are the premise that the existing fMRI decoding technology cannot be applied in actual brain-computer interface tasks. Therefore, it is necessary to design a framework that is suitable for EEG real-time decoding and online image reconstruction applications. For EEG data with low signal-to-noise ratio, non-stationarity, and possible time window offset, it is impossible for EEG to be as simple as fMRI encoder. It requires special design to meet the requirements of replacing manual features. In terms of the superiority of our framework, compared to fMRI-based decoding that can only be attempted on datasets, our proposed EEG encoder can directly process EEG signals and respond quickly to complete a series of decoding and image reconstruction tasks.

In order to demonstrate the competitiveness of our framework, we used different neural encoder to test the reconstruction performance on the THINGS-EEG and THINGS-MEG dataset and the results are shown above.

Q4. Is the spider plot in Figure 4 showing normalized performance?

As shown in Figure 4, in each metric direction, we use the accuracy calculated by the best performing method as the benchmark and draw a normalized radar chart to more intuitively show the superiority of our method. Its absolute performance can be seen in Table 7 in the Appendix H.1.

Reference

[1]Benchetrit Y, Banville H, King J R. Brain decoding: toward real-time reconstruction of visual perception[C]//The Twelfth International Conference on Learning Representations.

[2]Fatma Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using generative latent diffusion. Scientific Reports, 13(1):15666, 2023.

[3]Paolo Scotti, Arpan Banerjee, Jessica Goode, et al. Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024.

[4]Takagi Y, Nishimoto S. High-resolution image reconstruction with latent diffusion models from human brain activity[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 14453-14463.