MindGPT: Interpreting What You See with Non-invasive Brain Recordings

Thank you very much for your valuable feedback and suggestions.

1. About the comparison. Many thanks for pointing out the potential competitors for comparison. We did not compare with the approaches because they were not officially published nor provided source-code, at the time we submitted. Following your suggestion, we have added comparative experiments (as well as a discussion) with UniBrain [1], a recent preprint on arXiv, which demonstrated the strength and advantage of our approach (please refer to the Related Work and Appendix). We only included one subject currently due to time limitation and the experiment with all the subjects is running and will be added to the results. Many thanks.

2. About the voxel vectors. The value of H is 3745, determined by the maximum dimension among the voxel vectors, where the sizes of V1-V4, LOC, FFA, PPA are 2036, 2291, 2223, 1851, 3745, 980, and 908, respectively. The amount of zero padding equals H minus the size of each voxel vector.

3. The details of MRI data preprocessing. Data samples were created by first regressing out nuisance parameters from each voxel amplitude for each run. After that, voxel amplitudes were normalized relative to the mean amplitude of the initial 24s rest period of each run and were despiked to reduce extreme values. The voxel amplitudes were then averaged within each 8s (training) or 12s (test) stimulus block, after shifting the data by 4s to compensate for hemodynamic delays.

4. About the few-shot reconstruction. 150 training fMRI-image pairs across 150 classes were removed from the training set and added to the test set. At the stage of training, we did not provide any information about the correspondence between these signals and their stimuli.

5. The results of Tabs 1 & 2. In the initial-version paper, we use subject 3 of the DIR dataset. We have added extra experiment on a large-scale fMRI dataset NSD in the revised paper (please see Appendix).

6. The role of linear projection. Yes, the linear projection is shared across all items in the sequence (i.e., the input of the fMRI encoder). Due to the zero-padding operation and the low signal-to-noise ratio inherent in fMRI data, the input contains a considerable amount of redundant information. The linear projection of the fMRI encoder plays a role in dimensionality reduction, which helps the model focus more on important features and assists it in learning higher-level semantic representations, thus aiding the model in generalizing better to new, unseen data.

7. Regarding the examples of Figure 3. For the image selection in Fig. 3, we showed the images commonly used for qualitative comparison following the prior visual reconstruction studies, to compare with them intuitively. The ratio of high-quality reconstructions to failure cases is approximately 1:2.5.

8. About the data augmentation. The original DIR contains 1,250 images. We extracted all the images across 150 training classes from ImageNet as the candidate set for data augmentation. In other words, we relaxed the original one-to-one relationship to one-to-many. This practice not only provides more semantic supervision signals (image captions in the same category) for fMRI recordings, but also mitigates the effects of some pseudo labels with low confidence. We have added an ablation experiment to evaluate the importance of data augmentations.

9. About the analysis of Section 4.4. We appreciate your valuable and constructive advice. By utilizing the actual CLIP [CLS] token instead of the decoder’s prediction for computing similarity scores, we noted a significant discrepancy from the predictions made by the MindGPT. These findings have been integrated into the revised version.

Many thanks for your constructive feedback, which significantly enhanced the overall quality of the paper. We have incorporated all the suggested modifications into the new manuscript. We hope our response can address your concerns. Would you please consider raising the scores?

References

[1] Mai, Weijian, and Zhijun Zhang. "Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity." arXiv preprint arXiv:2308.07428 (2023).

Thank you very much for your valuable feedback and suggestions.

1. Regarding the NSD dataset. The reason we did not use the NSD was that the MS-COCO dataset contains detailed annotations for images, but it does not provide specific classification labels like ImageNet. In other words, semantically complex scenes make it difficult for us to establish proper correspondences between virtual neural signals and candidate images. On the contrary, the images of ImageNet (typically including a single object) allow us to exploit virtual fMRI-image correspondences under the same class semantics. We do agree that NSD dataset has advantages over DIR dataset in terms of the scale (10k images) and semantics richness. Therefore, we have added an extra experiment with NSD dataset in comparison of a preprint work [3] (also used NSD). We only included one subject currently (please see Appendix) due to time limitation and the experiment with all the subjects is running and will be added to the results. Many thanks.

2. About the ablation study. We have added an ablation experiment to evaluate the importance of data augmentation, which has been incorporated in the Appendix of the revised paper. Many thanks.

3. About the quantitative results. The results in the paper are obtained via calculating language similarity metrics between the pseudo-caption (provided by SMALLCAP) and the generated text of MindGPT. We agree that this is a potential limitation of the approach. During the training phase, we used 200.7k candidate images for augmentation, making it challenging to collect human captions, but we are working in that direction.

4. The discussion about relevant references. Previous works [1-4] mapped fMRI recordings to the embeddings of pre-trained neural networks like VGGNet by relatively simple linear regression models, and then feeds the predicted result into language models to generate word sequences. Unlike existing decoding paradigms, our MindGPT is designed to explore self-supervised neural semantics learners by using cross-attention mechanisms and data augmentation techniques. To the best of our knowledge, generating linguistic semantic information directly from a single brain image in an end-to-end fashion has not been adequately explored. The above discussion has been incorporated into the revised version. Many thanks.

5. About the visual cues. The visual cues demonstrated that the model autonomously acquired information about the salient regions within an image, a process devoid of any explicit labels or supervision. By utilizing the actual CLIP [CLS] token for computing similarity scores, we noted a significant discrepancy from the predictions made by the MindGPT. These findings have been integrated into the revised version. Our preliminary hypothesis is that the visual cues are related to the subjects' eye movements when viewing the images. Comparing them (i.e., attention maps) to eye movement data may provide a more convincing answer. We think it would be a valuable research topic in our future studies.

6. About the weakness. The discussion of the limitations, e.g., the upper bound of accuracy, of this work has been incorporated into the revised version. Many thanks.

7. The Captions of Figs 3, and 4. Blue color text in Figs 3, and 4 indicates correct (or semantically similar) captions and incorrect text is highlighted in red. We have added explanations in captions of Figs 3, and 4.

Thanks for the valuable comments, which we do believe help improve the accessibility of the paper. We have incorporated all the suggested modifications into the resubmitted manuscript. We sincerely hope our response can address your concerns. Would you please consider raising the scores?

References

[1] Matsuo, Eri, et al. "Describing semantic representations of brain activity evoked by visual stimuli." 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2018.

[2] Takada, Saya, et al. "Generation of viewed image captions from human brain activity via unsupervised text latent space." 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020.

[3] Mai, Weijian, and Zhijun Zhang. "Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity." arXiv preprint arXiv:2308.07428 (2023).

[4] Ferrante, Matteo, et al. "Brain Captioning: Decoding human brain activity into images and text." arXiv preprint arXiv:2305.11560 (2023).

Thank you very much for your valuable feedback and suggestions.

1. About the novelty. We proposed a novel idea to “mind reading” by interpreting brain signals into words. From the perspective of method, the data augmentation and cross-attention was the key technical component for attaining the goal, since the main challenge was how to extract the semantic component from limited fMRI data. In other words, our final goal is to guide latent neural representations towards a desired language semantic direction, instead of alignment with CLIP visual space. Unlike existing decoding paradigms [1-4], using simple linear regression models, our MindGPT is designed to explore self-supervised neural semantics learners by using cross-attention mechanisms and data augmentation techniques. We have modified our expressions in the paper to make the novelty clear. Many thanks.

2. About the evaluation. To best of our knowledge, there hasn't been any similar open-source work on brain-to-text tasks published in peer-reviewed journals or conferences. To further validate the effectiveness of our approach, we have included comparative experiments with UniBrain [3] (using CLIP model), which is a recent preprint available on arXiv (please see Appendix).

3. About the details of this work. a) Subject: In the initial-version paper, we use subject 3 of the DIR dataset. We have added extra experiment on a large-scale fMRI dataset NSD in the revised paper (please see Appendix); b) Alignment: During the training process, we applied a CLIP-based L2 constraint to regulate the fMRI embeddings; c) fMRI data preprocessing: Data samples were created by first regressing out nuisance parameters from each voxel amplitude for each run. After that, voxel amplitudes were normalized relative to the mean amplitude of the initial 24s rest period of each run and were despiked to reduce extreme values. The voxel amplitudes were then averaged within each 8s (training) or 12s (test) stimulus block, after shifting the data by 4s to compensate for hemodynamic delays.

Thanks for the constructive comments, which helped improve the paper’s quality. We have incorporated all the suggested modifications into the new manuscript. We sincerely hope our response can address most of your concerns. Would you please consider raising the scores?

References

[1] Matsuo, Eri, et al. "Describing semantic representations of brain activity evoked by visual stimuli." 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2018.

[2] Takada, Saya, et al. "Generation of viewed image captions from human brain activity via unsupervised text latent space." 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020.

[3] Mai, Weijian, and Zhijun Zhang. "Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity." arXiv preprint arXiv:2308.07428 (2023).

[4] Ferrante, Matteo, et al. "Brain Captioning: Decoding human brain activity into images and text." arXiv preprint arXiv:2305.11560 (2023).

Thank you very much for your valuable feedback and suggestions.

1. About visual reconstruction. Our goal is to interpret the scene we see rather than reconstruct the image. The underlying hypothesis is that our brain is not a camera, and we can’t actually store all the pixels we see. Instead, we process visual information (at a semantic level) in our brain. Thus, we think that decoding semantic information and expressing it linguistically could be a more efficient way to understand the brain.

2. The discussion and comparison with other competitors. Decoding semantic sentences instead of reconstructing the image is a novel perspective and to best of our knowledge, there hasn’t been any similar open-source study officially published at the time we submitted. To validate the effectiveness of our approach, we have added comparative experiments (as well as a discussion) with UniBrain [1], a recent preprint on arXiv, which demonstrated the strength and advantage of our approach (please refer to the Related Work and Appendix).

We sincerely hope our response can address most of your concerns. Would you please consider raising the scores?

References

[1] Mai, Weijian, and Zhijun Zhang. "Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity." arXiv preprint arXiv:2308.07428 (2023).