fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding

Thanks for your time reviewing and providing professional feedback for our manuscript. Our codes and models will be released, and here are responses to specific questions.

1. The research goals are often unclear or framed in a general way

Thanks and we apologize for confusions. Following previous literature [A,B], we use the same word of 'brain activity decoding' to define our task. Also, we focus on visual cortical regions for a fair comparison. As for surface data, it is one of our contributions. Different from previous work that directly uses flattened 3D voxels of fMRI data as input, we additionally transform 3D voxels to 2D surface images and take them as a good carrier of information.

2. More quantitative analyses should be performed

Thanks for raising this question. As indicated by the title of subsections, our ablation studies are discussed in Sec, 4.2.1 and 4.2.2, as well as Tab. 1. More concretely, 'LR' and 'LEA' are two important baselines, where the former excludes the pre-trained encoder and the latter only has one stage for fMRI pre-training. Experiments in Tabs. 1, 2 and 3 clearly demonstrate the effectiveness of our proposed modules.

3. How fMRI-PTE differs from masked auto-encoder architectures?

Thanks, we explain that our method and MAE both use autoencoders for pre-training. Differently, we reconstruct the original images from the compressed features, but MAE adopts the masked modeling strategy. To verify the efficacy of two-stage architecture, we compare with LEA, a competitor that only takes one-stage architecture for reconstruction. Moreover, we make a slight modification to LEA to ensure a similar number of parameters and training iterations. Under this circumstance, results in Tabs. 1~4 clearly demonstrate the superiority of our design.

4. Innovative approach to transform fMRI signals

We apologize for any misleading. As explained in Sec. 3.1, we adopt the off-the-shelf toolkit to transform 3D voxels of fMRI data to 2D surface images. We acknowledge that it is a common preprocessing in neuroscience. However, taken it as a good carrier of information and further explore brain decoding on it is novel. Compared with previous method that mostly takes 1D vectors of voxels as input, experiments in Sec. 4.2 and 4.3 have demonstrated the effectiveness of this representation of fMRI data. We will revise the description and tone down the claim, making them more precise.

5. Pipeline in Fig. 1 is not meticulously depicted

Thanks for your suggestion. We will polish the figure and add more descriptions. For your specific questions, (1) during pre-training, we use fMRI data only from UKB dataset. (2) GPT refers to decoder-only transformers, which serves as the decoder $\mathcal{D}_{2}$ in Sec. 3.2.2. (3) the 'fires' and 'ices' symbols represent whether model parameters are learned or fixed during training.

6. More details about vector quantization and corresponding loss terms

Thanks for the question. As stated in Sec. 3.2.1, the architecture and training objectives are the same as VQGAN [C]. Thus, we refer readers to [C] for more details. We will add the footnote to make it more clear in the revision.

7. Why is a Vision Transformer required to compress the quantized indices?

Thanks. Without feature compression, the autoencoder performs more like an identity mapping, passing 100% information from the encoder to the decoder. It would significantly improve the quality of reconstruction, but not help the encoder's learning.

8. Is input the entire cortical image?

No, our input for both pre-training and brain decoding task is surface image only with visual ROIs. Concretely, we transform the whole cortex to 2D images, and use the mask of visual RoIs to crop it. We will make it more clear in the revision.

9. Poor reconstruction accuracy of MAE?

Good comments. As explained in Sec. 4.2.1, we are also surprised by the poor reconstruction accuracy of MAE. We explain that possible reasons could be (1) less spatial redundancy in brain surface images compared to natural images; (2) large white background in brain surface images (note that our method also reconstructs the background); (3) more training efforts required for masked modeling strategy. In contrast, our method is tailored to brain surface images, showing its superiority.

10. Writing, acronyms and Typos

We apologize for any mistake. We will correct them and do careful proofreading multiple times in the revision.

References

[A] Takagi, Yu, and Shinji Nishimoto. "High-resolution image reconstruction with latent diffusion models from human brain activity." CVPR, 2023

[B] Chen, Zijiao, Jiaxin Qing, and Juan Helen Zhou. "Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity." NeurIPS, 2023

[C] Patrick Esser, Robin Rombach, and Bjorn Ommer. "Taming Transformers for High-resolution Image Synthesis". CVPR2022

Thanks for your time reviewing and providing professional feedback for our manuscript. Our codes and models will be released, and here are responses to specific questions.

1. Lacks a strong baseline of MindEye

Thanks. The main purpose of this paper is for cross-subject brain activity decoding. Experiments in Tab. 2 and 3 have demonstrated the advantage of our method. Like previous methods, MindEye is for within-subject brain decoding, and it needs to train from scratch for each subject. However, it is a good work that utilizes diffusion prior for fMRI-Image mapping, published in arXiv before the submission. We will cite and discuss it in the revision.

2. No zero-shot capabilities have been explore

Thanks for the question. We would like to clarify that,

• Our proposed fMRI pre-trained transformer encoder is mainly for cross-subject brain activity decoding. It would be interesting future work to explore its application to other tasks, like brain disease classification.
• In addition to cross-subject brain decoding task, we also reported results on within-subject task, achieving competitive or even better performance compared with end-to-end training competitors. It clearly suggests the generalization ability of our proposed method.
• Results in Tab. 2 and 3 exactly show the zero-shot capability of our method, where we use data from one or many subjects to align fMRI and image features, and directly evaluate the model on one unknown subject. The higher performance achieved by our method indicates its superiority

3. Innovative Data Transformation is not novel

We apologize for any misleading. As explained in Sec. 3.1, we adopt Connectome Workbench and pycortex toolboxes to transform 3D voxels of fMRI data to 2D surface images. We acknowledge that it is a common preprocessing in neuroscience. However, taken it as a good carrier of information and further explore brain decoding on it is novel. Compared with previous method that mostly takes 1D vectors of voxels as input, experiments in Sec. 4.2 and 4.3 have demonstrated the effectiveness of this representation of fMRI data. We will revise the description and tone down the claim, making them more precise.

4. Claims about preserving high-frequency signals are confusing

Thanks, but we would like to clarify some misunderstandings that,

• Low- and high-frequency refer to signals in 2D images, because we transform 3D voxels of fMRI data to 2D surface images.
• We use the autoencoder to reconstruct 2D surface images frame by frame, instead of performing the temporal reconstruction
• Based on the above clarification, we have evaluated the quality of reconstructed surface images with three image-based metrics in Tab. 1, showing the advantage of our method.

5. The MAE baseline is confusing and not comparable

Thanks, but we would like to clarify that,

• In addition to MAE, we also compare our method with two competitors, MBM and LEA, that specifically designed autoencoder for fMRI data.
• A higher masking ratio is the default setting of masked modeling strategy. Notably, MBM also uses the similar masked modeling strategy with a higher masking ratio.
• For a fair comparison, we made a slight adjustments to these baseline models to ensure a similar number of parameters and training iterations.

6. Details for the preprocessing of the fMRI dataset

Thanks. The UKB and NSD datasets used in our paper were already preprocessed into MNI and fsaverage group space and we only applied transformation between group spaces. According to the descriptions of UKB and NSD, they did apply motion correction.

All of these clarifications have been explained in the first paragraph of Sec. 4.2.1, so results in Tab. 2 are enough to show the effectiveness of our method.

7. Lack of variability in Tabs. 2, 3 and 4

Thanks. As stated in Sec. 4.1, we follow [A] to use several low-level and high-level metrics to measure the fidelity and accuracy of the generated images from fMRI signals. As a result, there are totally 8 metrics used in Tabs. 2, 3, 4, which are comprehensive and enough to evaluate the effectiveness of our method.

References

[A] Ozcelik, Furkan, and Rufin VanRullen. "Brain-diffuser: Natural scene reconstruction from fmri signals using generative latent diffusion." arXiv preprint arXiv:2303.05334 (2023).

Thanks for your time reviewing and providing professional feedback for our manuscript. Our codes and models will be released, and here are responses to specific questions.

1. The methodological innovation is marginal

Thanks for the question, but we would like to clarify some misunderstandings that,

• Both MBM and our method are proposed to build fMRI pre-trained autoencoder for brain decoding task, so they are directly comparable.
• Our method is not designed based on MBM. Though they have the same purpose of fMRI pre-training, they are quite different from two aspects, architecture and inputs. MBM takes as input 1D fMRI vectors, and use masked modeling [A] for pre-training. Instead, we take 2D fMRI surface images as inputs, and design two-stage learning strategy with dimensional compression for pre-training.
• Previous methods only focus on within-subject brain decoding. In this paper, our technological contribution lies not only in the fMRI pre-trained autoencoder, but also in the framework for both within- and cross-subject brain decoding. Experiments in Sec. 4.2 and 4.3 have shown the efficacy of our method.

2. 'Innovative data transformation' seems trivial

We apologize for any misleading. As explained in Sec. 3.1, we adopt the off-the-shelf toolkit to transform 3D voxels of fMRI data to 2D surface images. We acknowledge that it is a common preprocessing in neuroscience. However, taken it as a good carrier of information and further explore brain decoding on it is novel. Experiments in Sec. 4.2 and 4.3 have demonstrated the effectiveness of this representation of fMRI data. We will revise the description and tone down the claim, making them more precise.

3. 'Analysis of efficient training and learning strategy

Thanks for raising the question. We would like to clarify that,

• The purpose of this paper is to design a fMRI pre-trained encoder for brain activity decoding. Thereby, the main experiments are within- and cross-subject brain decoding, which have been conducted and evaluated in Sec. 4.2 and 4.3.
• To further investigate the efficacy of our pre-trained encoder, we compare the quality of reconstructed images with other methods in Tab. 2. Specifically, we deliberately made slight adjustments to competitors to achieve a similar number of parameters and the same number training iterations, to verify the advantage of efficient training of our method.

4. Lacks details of GPT and image decoder in the method part

Thanks and we apologize for confusions. (1) GPT refers to decoder-only transformers, which serves as the decoder $\mathcal{D}_{2}$ in Sec. 3.2.2. (2) In the last paragraph of Sec. 4.1, we discuss that the image decoder could be either stable diffusion or MaskGIT. We also show results with both decoders in Tabs. 2~4. We will polish the figure and add more explanations.

5. It would be beneficial to provide information about the kernel size

Good question. The kernel size we used is $3\times 3$, and we resize the input surface images to $256\times 256$, so it could capture information in small regions. In this paper, we process fMRI data as 2D images, the setting of kernel size is standard in computer vision. So, we don't tune it heavily.

6. The method for choosing cross-subject pairs is not clear.

Thanks. We explain that the combination of cross-subject is numerous, especially for the one-to-one setting. In Tab. 2, we randomly choose 4 representative pairs for evaluation, where both the source and target domain cover 4 subjects. Combined with results in Tab. 3, experiments are self-contained and enough to show the capacity and superiority of our method on cross-subject brain decoding task. We will add more explanation about pairing choices in the revision.

7. Batch size is mentioned, but no corresponding variable seems to be provided

We sorry for typos. In Sec. 3.2.2., words of 'batch size' should be deleted. It is for simplicity of symbolic representation, and we explain the number of batch size in Sec. 4.1.

8. Presenting the brain visualizations with a color scale

Good suggestion. We post-process fMRI surface images by min-max normalization and visualize them in Fig. 3. We will further add colorbar to improve the clarity of results.

References

[A] He, Kaiming, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. "Masked autoencoders are scalable vision learners." CVPR2022

Thanks for your time reviewing and providing professional feedback for our manuscript. Our codes and models will be released, and here are responses to specific questions.

1. Lack of ablation experiments to validate the proposed model

Thanks for your question. We would like to clarify that we have conducted the suggested study in Sec. 4.2 and 4.3. Concretely, as explained in the last point of Sec. 4.2.2, we present 'LR' as a baseline, where we serialize pixels of the surface map as vectors of fMRI for brain decoding. This baseline is as a variant of our model where it excludes the fMRI pre-trained encoder. Therefore, results in Tabs. 2-4 are clear and enough to show the effectiveness of our pre-training on UBK dataset. We will make it clearer in the revision.

2. The training process is not adequately described

Thanks. We totally understand this concern. We have elaborated implement details in Sec. 4.1, including optimizer parameters and two-stage training way, to make our work reproducible. Moreover, our codes and models will be released upon acceptance.

3. Abbreviations are not clearly defined

We sincerely apologize for any confusion and typo. We will revise them and do careful proofreading. For your specific questions, (1) 'GLM' means general linear model; (2) there is a typo, words of 'batch size' should be deleted; (3) $E_{2}$ is the encoder in the second stage, where it takes as inputs quantized feature indices (stated in the first paragraph of Sec. 3.2.2). The input dimension is $\in \mathbb{R}^{L}$, where $L$ is the length of indices, it is the same as the number of feature pixels output by the first stage encoder; (4) 'M', 'T' and 'V' in Tab. 4 indicate modality, text and vision, respectively.