Language Reconstruction with Brain Predictive Coding from fMRI Data

We sincerely appreciate your effort in reviewing our paper. We will address your concerns point by point.

Why did the author only use BLEU and ROUGE in the experiment?

We choose BLEU and ROUGE in the experiment to follow the evaluation setting in UniCoRN [1]. Generally speaking, BLEU, ROUGE, WER, METEOR all measure word-level overlapping between generated content and ground truth. Choosing any of them will lead to similar results, while BERTScore is designed to evaluate semantic-level similarity. We add the BERTScore result of different models here for supplementary.

Results show our model performs good in semantic-level reconstruction (the gap compared to the best is narrow), while maintaining the best BLEU and ROUGE score.

[1]UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language. ACL'23

Many of the methods compared by the author incorporate LLM, while the author's model is entirely trained with their own transformer. Does this result in the author's method being inferior to the baseline method in terms of semantic similarity?

The above supplementary BERTScore result confirms that our model achieves similar semantic reconstruction ability as LLM-based method. Moreover, the disadvantage of LLM-based method is that the parameters of LLM is fixed and can't be finetuned. Only the input of LLM could be changed and previous methods (Tang's model, BrainLLM, MapGuide) try different ways to improve input representation. While training Transformer based model from scratch is more flexible (allowing architecture changes).

The author's method was inspired by predictive coding and validated it on LLM using a prediction score. But can the author's own model still observe the same phenomenon on the prediction score? I haven't seen the same experiment evaluating the author's own model.

We have to clarify that they're two completely different types of tasks:

• Prediction score

  It belongs to brain-LM alignment research (also termed brain encoding in some paper). The common method is mapping (e.g. via linear model, rsa) the output of LM layer to brain response, and analyzing voxel-level correlation. It aims to discover whether LM has human-like language comprehension ability.

• Language reconstruction

  The task is also known as brain decoding, fMRI-to-text decoding. It aims to decode natural language from brain recordings.

As a result, it's meaningless to calculate the prediction score for PredFT because it's not a language model. Even if we do so, the output result lacks practical meaning. We present the prediction score experiment to highlight how we get the motivation of PredFT: while predictive coding improves brain encoding, could it help improve brain decoding?

In some parts of the paper, fMRI is spell as FMRI.

fMRI is written as FMRI only in Abstract to highlight how we name the model: the FT in PredFT comes of FMRI-to-Text.

We sincerely appreciate your effort in reviewing our paper. We will address your concerns point by point.

Weaknesses

Weakness 1

There are several major weaknesses in this work, particularly concerning the evaluation of reconstruction results. A major concern is that the current study (PREDFT) does not provide a clear evaluation of reconstruction results.

We have already provided a case analysis containing reconstruction results in Appendix F. If you are still confused about the reconstruction results, please refer to our reply to reviewer XUt4.

For example, the authors did not evaluate the word rate in the generated narrative story.

There's no word rate model in PredFT. We don't use a fixed LLM to endlessly generate decoded content like Tang's model, in which case a word rate model is needed to judge the end point. Instead, we view fMRI-to-text decoding as sequence-to-sequence translation task and train PredFT in an end2end manner. So how many words should be decoded is already learned by PredFT. We only need to set a sampling method (e.g. greedy decoding).

Weakness 2

However, prior studies have focused on specific ROIs, such as the language, prefrontal, and auditory association cortices...

In the main decoding network, we apply whole brain data for reconstruction instead of selecting specific ROIs related to language comprehension. In the side network, we use BPC area for language reconstruction, which covers most part of language related areas like auditory cortex (AC), prefrontal cortex (PFC) and Broca area.

The random selection of ROIs generally leads to low decoding performance. What are these random ROIs? Do they have any overlap with BPC ROIs?

The specific ROIs we applied are listed in Appendix A.4. For example, for narratives dataset G_and_S_cingul-Ant, G_and_S_subcentral, G_and_S_transv_frontopol, G_orbital, S_front_middle, S_subparietal are selected. For LeBel's dataset, we randomly choose 1000 voxels from brain surface data besides BPC area(because fMRI in this dataset isn't projected to a standard space (e.g. MNI space), so we can't apply a brain atlas for parcellation). There is no overlap between Random ROIs and BPC ROIs.

this paper does not report any reconstructed stimulus in the main content, nor does it include analysis at the ROI level.

The reconstructed stimulus is shown in Appendix F. Since we use whole brain data in main decoding network for language reconstruction, there's no available ROI level reconstructed stimulus. We also notice that only Tang's work conducted ROI level reconstructed stimulus, while other previous works (BrainLLM, unicorn, mapguide, BP-GPT) didn't include it.

Additionally, the authors only used two metrics, and throughout the paper.

We add another metric BERTScore. Please refer to rebuttal part 2.

Weakness 3

there are no qualitative reconstruction results for these different predictive lengths and distances.

When the prediction length and distance is too long or short, it becomes distraction and the decoded content becomes meaningless. No essential concepts or semantics can be decoded. So we think there's no need to show them. The reconstruction results with proper length and distance are shown in Appendix F.

What type of information is the model forecasting based on brain data?

The training objective for side network is cross-entropy loss between generated words and label, and the label is ground truth word sequence with certain prediction length and distance from the onset word of each fMRI frame. So the prediction is supposed to be the combination of syntactic and semantic information. We can't show what exactly the model predicts since the predictive information is presented in the form of representation, which is a deep learning approach. We can only judge whether it works through reconstruction performance (e.g. bleu).

Weakness 4

All the figures lack detailed captions.

The explanation for concepts like "prediction score", "prediction distance" is presented in the first paragraph of section 2. We will add more captions in the edited version of paper for better understanding.

Weakness 5

it is unclear which component is primarily responsible for reconstructing the language and which component provides the theme and narrative structure.

The main decoding network in PredFT is responsible for whole text reconstruction, while the side network provides predictive coding information and the decoder in side network is discarded after training. Therefore, no specific part is claimed to be responsible for reconstructing the language, nor is any particular part claimed to provide the theme and narrative structure, the decoder in main decoding network generates as a whole.

Questions

Q1

What would be the chance-level performance when reconstructing continuous language?

We are conducting a chance-level baseline, please lend us more time.

what is the percentage of overlap between random ROIs and whole-brain voxels?

None overlapping.

Did the authors repeat the selection of random ROIs multiple times to ensure robustness?

Yes, the selection of random ROIs is repeated five times and the results are similarly poor.

Q2

What is the rationale for using 4D volume data from the Narratives dataset while using 2D brain data from the Moth Radio Hour dataset?

We selected 4D volumetric whole-brain data from the Narratives dataset to align with the settings used in Unicorn, allowing us to compare performance of different models.

Q3

There is no interpretation provided for the two encoders used in PREDFT.

Thanks for raising this problem. We think directly evaluating the quality of encoder representation is hard, since we apply a deep learning approach instead of a linear encoding model. Using the change of decoding performance to judge the quality of encoder representation is an alternative and meaningful approach. Project the representation to brain map doesn’t seem to lead to results with practical meaning. Previous works didn’t test the encoding quality in this way either.

Q4

Figures 3, 4, 6, and 8 appear redundant.

We hope showing different components separately can help readers understand the model easily. We also provide the whole framework of PredFT in Figure 11 in the Appendix.

Q5

What does the y-axis represent in Figure 9?

Y axis represents the numerical value of score in a percentage-based system (e.g. 40 means 40%). We will polish the figure.

Typos

Thanks for pointing out. We will fix it.

BERTScore supplementary

In the regions of interests selection experiment, the authors only consider 'random,' 'whole,' and 'BPC' as the ROIs...

The selected BPC area (superior temporal sulcus, angular gyrus, supramarginal gyrus, and opercular, triangular, orbital part of the inferior frontal gyrus) contributes the most to predictive coding, as indicated in some neuroscience studies [1][2].

The process for selecting "BPC": For the Narratives dataset, Destrieux atlas is applied and the above mentioned ROIs are extracted. For LeBel's dataset, since the fMRI signals are not projected to a standardized space, we use the “Auditory” region provided by the authors', containing parietal-temporal-occipital (PTO) area. The BPC area of both datasets cover highly similar area.

The process for selecting "random": For the Narratives dataset, G_and_S_cingul-Ant, G_and_S_subcentral, G_and_S_transv_frontopol, G_orbital, S_front_middle, S_subparieta are selected. For LeBel's dataset, we randomly choose 1000 voxels from brain surface data.

The process for selecting "whole": We use the whole brain surface data as ROIs for both datasets.

We believe selecting random and whole ROIs as controlled experiments is sufficient for demonstrating the effectiveness of using predictive coding to improve decoding performance:

1. random vs. BPC demonstrates only ROIs related to predictive coding in human language comprehension can improve decoding.

2. whole vs. BPC not only confirms conclusion in 1, but also shows whole brain surface which contains BPC area still can't contribute to better decoding, because some other brain regions contain too much noise.

3. none (PREDFT without SideNet) vs. BPC. PREDFT without SideNet is equivalent to not using any ROIs for predictive coding. This comparison shows predictive coding improves decoding accuracy significantly.

All the above clarifications are included in sec 4.3 and Appendix A.4. We will add more key information in sec 4.3 in the updated version

[1]. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature Human Behavior [2]. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature

Could the authors provide pseudocode for the method...

We will provide pseudocode in the appendix for edited version of paper. The discussion of time complexity is already in Appendix A.4 (line 845) of original paper.

The results provided by the authors mostly only include the mean value...

We guess you indicate the experiment of analyzing the impact of prediction length and distance to model performance (sec 4.4), as we have presented results per subject for other experiments. The per-subject results of analyzing the impact of prediction length and distance are already presented in Figure 16,17,18 in the appendix of original paper.

In the methods section, some symbols are not defined...

A notation table for symbols is presented in Table 3 in the appendix of original paper.

Questions

Please refer to clarification for weaknesses.

We sincerely appreciate your effort in reviewing our paper. We will address your concerns point by point.

Weaknesses

In Section 3.3, the authors state, 'During the inference stage, as illustrated in Figure 8, the decoder in the side network is abandoned.' However, they do not provide a detailed explanation of why the decoder is discarded or discuss the potential impact of this decision...

What we really want is predictive coding representation, which is produced by side network encoder. The side network decoder is designed to help train the side network encoder (we can't find out how to directly obtain predictive coding representation). Specifically, during the training process, the label for side network decoder is predicted words instead of complete sentences (as shown in Figure 3). The side network learns mapping between specific areas of brain (BPC area) and predicted words. However, the goal of our task is to decode complete sentences. So the side network decoder is useless after training.

We will add the motivation and reason for discarding the side network decoder in the methodology section in the updated version.

As shown in Table 1, PREDFT does not achieve the best performance on ROUGE1-R...

We think the different lengths of generated content might contribute to this factor, since we don't apply a word rate model to control the number of generated words. Although PREDFT fails to outperform other models in ROUGE1-R, the gap is narrow. Just like recall and precision to f1 score, ROUGE-Recall measures the extent to which a machine-generated content captures the information contained in a reference content, which is a single perspective assessment. ROUGE-Recall and ROUGE-Precision are characterized by a trade-off. When PREDFT gets a relative low ROUGE-R, it gets a high ROUGE-P. Instead, ROUGE-F1 is a more comprehensive indicator, combining both ROUGE-P and ROUGE-R. Our model outperforms other models in this metric.

As shown in Table 1, PREDFT without SideNet performs similarly to other methods. However...

The SideNet is designed to obtain predictive coding representation. PREDFT without SideNet can be viewed as traditional deep learning approach which directly applies Transformer to decode text from brain recordings, while PREDFT with SideNet combines deep learning and neuroscience findings (predictive coding). The SideNet provides predictive coding representation to the decoder in Main network, and the decoder incorporates both current fMRI representation and predictive coding representation for text decoding.

The idea of PREDFT is motivated by predictive coding theory in Neuroscience, which indicates human can naturally predict upcoming words. Since predictive coding has been verified to contribute to human language comprehension, we seek to investigate whether such predictive information can help language reconstruction. The improvement of incorporating SideNet highlights 1. the effectiveness of our model design 2. predictive coding has potential to improve brain-to-text decoding. We will provide the illustration of PREDFT without SideNet for better understanding in the updated version.

Although the authors provide a detailed description of the hyperparameter selection...

All the hyperparameters are chosen to minimize the training & validation loss as much as possible. We don't understand which hyperparameter the reviewer is confused about. The influences of ROIs selection, prediction length, prediction distance, $\lambda$ to model performance are detailedly discussed in sec 4.3, sec 4.4, Appendix E. The learning rate is set to stabilize training. We don't test the influence of model layers (e.g. Transformer layers) due to limited computational resources.

We add BERTScore of different models here.

For LeBel's dataset:

For Narratives dataset:

We sincerely appreciate your effort in reviewing our paper. We will address your concerns point by point.

Unconvincing reconstruction result.

We fully understand your concern. We noticed similar problems during experiments, and here's our explanation.

fMRI-to-text decoding is an important but extremely difficult task due to 1. noisy fMRI signal with latency 2. mismatch between fMRI sampling frequency (2s) and word rate (0.2s). We haven't found any models that can decode satisfying text. If you detailedly read the cases (not cherry picked cases) in previous work (e.g. page S2 in Tang's paper https://www.biorxiv.org/content/10.1101/2022.09.29.509744v1.full.pdf ), you will find that, although the generated content is locally coherent, it's far from being relevant to the ground truth. We present the decoding results of the first 10 seconds from Tang's paper here for example:

S3 is considered as good case by the authors, while it only decodes the concept of "crying" and "purse". Not to mention the content of S1 and S2 is completely irrelevant. Our model can decode more concepts as shown in our random picked cases. So the question becomes: Could it be said that a coherent but completely irrelevant text is preferable to an incoherent text but containing more important concepts in fMRI-to-text decoding task? We believe using metrics to evaluate different models is fair and reliable. BLEU has several drawbacks as you mentioned, which makes BLEU-1 less convincing. But BLEU-3, 4 is reliable. Because there're not many meaningless and repeated trigrams in ground truth, so high BLEU-3,4 prove that model decodes more important concepts. We believe the 5.62 BLEU-3 just shows superiority of our model. Besides, we supplement BERTScore to evaluate semantic similarity for more convincing results in Rebuttal (part 2). Results show our model performs good in semantic-level reconstruction (the gap compared to the best is narrow), while maintaining the best BLEU and ROUGE score.

Moreover, we have identified the reasons that lead to this phenomenon. We divide existing models into two categories:

1. The parameters in LM are fixed (Tang's method, BrainLLM, MapGuide).

• Advantages

  • Can generate coherent text with the power of pre-trained LM.

• Disadvantages

  • This approach largely restricts model architecture. Only the input representation can be changed. The innovation of previous works is that they try various ways to enhance input representation.

2. The parameters in LM are changed (Unicorn, PredFT)

• Advantages

  • The model design is flexible without restriction. Many innovations might be sparked.

• Disadvantages

  • It's hard to maintain good coherence when finetuning LM or training from scratch due to the very complex relationship between fMRI and text (as mentioned in the beginning).

Both types of models can decode part of important concepts.

In conclusion, while we still have a long way to go towards coherent and accurate fMRI-to-text decoding, we hope the above clarification convinces you that our work represents a solid step.

The paper is also quite difficult to read for no reason and pointlessly notational...

The three attention equations represent different meanings:

• Self-attn: q, k, v all come from the input to self-attention layer. It captures relationship between words.

• ED-attn: q comes from the input to self-attention layer, k and v come from the output from main network encoder. It captures relationship between fMRI and words.

• PC-attn: q comes from the input to self-attention layer, k and v come from the output from side network encoder. It captures relationship between predictive information and words.

We hope readers can have a better understanding of how PredFT operates (especially the input of different attention blocks) with the presented equations. We will only keep the predictive coding attention (PC-attn) equation in the edited version if equations become distractions.