Brain-Inspired fMRI-to-Text Decoding via Incremental and Wrap-Up Language Modeling

We appreciate your valuable feedback, and our detailed responses are provided below.

(1) W1&Q3 Examples of Qualitative Results： Due to page limitations, we provide the qualitative analysis results (i.e., actual text decoding examples) in the supplementary materials, specifically in Section A.3 (“Text Cases”) of the appendix. In this section, we present representative decoding examples of our method and SOTA approaches under different input lengths. The results show that our approach outperforms SOTA methods in capturing both semantic and syntactic information at the token level. It can identify more key content words, such as verbs, nouns, and named entities like person and place names. The generated sentences are also more semantically aligned with the reference texts. In the final version, we will include some of the decoded text in the main body of the paper.

(2) W2 Ablation of Sequential Decoding： Following your suggestion, we added an ablation study to evaluate the effectiveness of Sequential Decoding. Specifically, we retained the text-guided masking module and the pretraining module, but replaced our proposed Sequential Decoding (i.e., segmented processing combined with incremental decoding and the wrap-up mechanism) with a holistic decoding approach. The new results, shown in the 4th row of the table below, demonstrate a clear performance drop when Sequential Decoding is removed. For example, BLEU-1 decreases from 38.3 to 21.6, ROUGE-R drops from 36.5 to 25.0, and BERTScore-R decreases from 55.1 to 47.2. These findings highlight that Sequential Decoding plays a crucial role in improving the decoding performance, validating its importance in our proposed framework. This ablation result will be added in our final version.

(3) Q1 Motivation of Segmental and Incremental Mechanisms based Method： fMRI-to-text decoding is fundamentally different from traditional machine translation tasks. Unlike translation, which performs a one-to-one mapping between two independent modalities, fMRI-to-text decoding reconstructs textual content from neural activity patterns generated during language comprehension of the human brain. As a result, models designed for cross-modal translation are not directly applicable to this task. Based on this insight, we hypothesize that adopting a decoding framework more aligned with human language processing mechanisms could better address this challenge. To this end, we propose a brain-inspired sequential decoding paradigm that incorporates both incremental processing and segmental integration. Both our experimental results (Section 4.3, Table 1) and ablation studies (Section 4.4, Table 2) strongly support this hypothesis. Compared with existing SOTA methods, our approach shows significant overall improvements, with particularly strong performance on long-sequence decoding tasks. The ablation studies further demonstrate that Sequential Decoding (i.e., segmented processing combined with incremental decoding and the wrap-up mechanism) contributes the most to performance gains, highlighting the importance and effectiveness of incorporating cognitively inspired components in fMRI-to-text decoding. In the final version, we will add a clearer explanation of the motivation behind segmental and incremental mechanisms.

(4) Q2 More Results on Longer fMRI： Due to the limited availability of fMRI-text paired data, increasing the fMRI sequence length further cannot provide a sufficient number of training samples to meet the requirements for effective model training. For example, when the fMRI length is increased to 80 TRs, the total number of samples is only around 1,600. Therefore, in our main text, we report results only for 20, 40, and 60 TRs. Following the reviewer’s suggestion, we have also included decoding results for fMRI signal with 80TRs. We observed performance degradation across all metrics. However, where the drop is caused by the model’s limitation in handling longer sequences or by the lack of sufficient training data still needs further exploration. This question can be further investigated in future work when more data becomes available.

We appreciate your valuable feedback, and our detailed responses are provided below.

(1) W1&Q1 Evaluation on Other Dataset： We further evaluated our method on an additional public dataset (LeBel et al., 2023), which includes fMRI recordings from 8 participants when they listened to 27 full-length natural narrative stories (totaling about 6 hours). The experimental configuration follows the same setup as used for the Narratives dataset in the study. The results for the 60TR decoding task are presented in the table below. As shown, our method also consistently achieves superior performance compared to current SOTA methods. These results will be added to the supplementary materials of our final version.

[1] LeBel et al., 2023. A natural language fMRI dataset for voxelwise encoding models. Scientific Data, 10(1): 555.

(2) W2&Q4 Theoretical analysis of why segmented processing outperform holistic approaches： Segmented processing offers clear theoretical advantages over holistic approaches, which can be explained from both cognitive science and modeling perspectives. From a neuroscience perspective, our segmented and sequential decoding framework that integrates incremental decoding and the wrap-up mechanism is consistent with numerous findings on how the human brain processes language. Prior studies have shown that humans typically divide linguistic input into smaller, manageable units (e.g., clauses or sentences) and integrate information at natural boundaries (Tiffin-Richards et al., 2018). This strategy effectively reduces cognitive load and improves comprehension accuracy. Behavioral evidence further indicates that when text passages are approximately 55 words long, humans achieve optimal reading efficiency and comprehension at both normal and fast reading speeds (Dyson et al., 2001). The optimal segment length identified in our model, 20 TRs (about 60 words), closely aligns with these cognitive findings, providing strong theoretical support for our brain-inspired segmented decoding design. From a modeling perspective, fMRI signals inherently have a low signal-to-noise ratio, and holistic decoding of long time series significantly increases the difficulty of extracting meaningful information. Inspired by the advantages of state-passing approaches for handling long sequences, we divide the fMRI time series into multiple segments, perform incremental decoding on each segment, and use the wrap-up mechanism to pass a semantic summary of the previous segment to guide the decoding of the next. This design not only mitigates performance degradation caused by excessive sequence length but also improves semantic consistency across segments. Comparative experiments with existing holistic decoding approaches (e.g., UniCoRN, EEG-Text, BP-GPT) demonstrate that our segmented decoding method achieves significantly better performance, further validating its effectiveness and practical advantages.

[1] Tiffin-Richards et al., 2018. The development of wrap-up processes in text reading: A study of children’s eye movements. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(7): 1051. [2] Dyson et al., 2001. The influence of reading speed and line length on the effectiveness of reading from screen. International Journal of Human-Computer Studies, 54(4): 585-612.

(3) W3&Q3 Over-Simplicity and Sensitivity of the Wrap-up Mechanism： Our proposed wrap-up mechanism is designed to semantically compress the textual information decoded from the previous fMRI segment and pass it as a summary vector to guide the decoding of the next segment. Thus, it is not a simple state propagation but rather a cross-modal semantic integration process involving modality translation and higher-level abstraction. Specifically, the mechanism first decodes an fMRI segment into text, encodes the generated text using BERT to obtain a rich semantic representation, and then projects this representation into the fMRI latent space of the subsequent segment via an MLP to assist in decoding. This design achieves semantic summarization and cross-modal transfer in a lightweight manner, avoiding the introduction of excessive parameters that could lead to unstable training. We systematically evaluated different wrap-up designs and found that more complex models do not necessarily yield better performance. For example, in experiments with varying MLP hidden dimensions, a dimension of 128 achieved the best performance on both word-level and semantic-level metrics, whereas increasing the dimension to 256 led to a significant drop in performance, even falling below that of 32 or 64 dimensions (Section 4.2, Figure 5). We also explored more sophisticated frameworks, such as integrating BLIP for semantic aggregation (Li et al., 2022). However, experimental results showed that the BLIP-based approach achieved comparable performance to the MLP-based approach but exhibited less stable training, likely due to its higher data requirements. Based on these findings, we ultimately adopted the simpler and more stable BERT+MLP design. In future work, we plan to explore more efficient wrap-up mechanisms to achieve better semantic compression and further improve model performance.

[1] Li et al., 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning 2022 Jun 28 (pp. 12888-12900).

(4) Q2 Fixed Segmentation scheme： The primary goal of this study was to validate the feasibility of the proposed brain-inspired fMRI-to-text decoding framework, particularly to examine whether the incremental decoding and wrap-up mechanisms can improve decoding performance for long text sequences. Therefore, in the current work, we adopted a relatively simple segmentation strategy with a fixed segment length. Through systematic experiments, we verified that using a segment length of 20 TRs (approximately 60 words) yields the best decoding performance. Interestingly, this result is consistent with findings from cognitive science showing that humans achieve optimal comprehension when reading passages of around 55 words (Dyson et al., 2001). This consistency provides theoretical support for the fixed segmentation strategy adopted in our study. As discussed in our paper, we acknowledge that a fixed segmentation strategy may split the narrative at unnatural points, resulting in improper semantic segmentation. Determining how to segment text effectively is a challenging but important topic. In future work, we plan to explore content-adaptive segmentation methods that dynamically predict segment boundaries based on narrative complexity or syntactic structure, enabling the model to flexibly adapt to diverse textual inputs. Furthermore, we aim to investigate approaches that combine dynamic segmentation with more flexible cross-segment information integration to achieve a better balance between local decoding accuracy and global semantic coherence.

[1] Dyson et al., 2001. The influence of reading speed and line length on the effectiveness of reading from screen. International Journal of Human-Computer Studies, 54(4): 585-612.

(5) Q5 Ablation of text-guided masking： In Section 4.4, we conducted an ablation study to verify the effectiveness of text-guided masking. Specifically, we retained the pretraining module (HCP dataset pretraining in the first stage) and the sequential decoding module (i.e., incremental and wrap-up based fMRI-to-Text decoding framework) while removing the text-guided masking strategy, and then evaluated the model’s performance (see the 3th row in the table below). To enhance the semantic relevance of learned fMRI representations, the proposed text-guided masking strategy identifies words with high semantic relevance, then prioritizes the reconstruction of fMRI time points associated with these key words, which directs the model’s attention toward fMRI signals associated with highly informative semantic content, thereby improving the semantic relevance and effectiveness of fMRI representations. The results show that removing text-guided masking led to a decrease in decoding performance across all metrics, further demonstrating the importance of this module in improving the model’s performance.

Dear Reviewer 4tPt,

We sincerely appreciate the time and effort you have devoted to reviewing our submission. We have carefully prepared our rebuttal, aiming to address the valuable points you raised in a direct and respectful manner.

Should you have an opportunity to review our response, we would be grateful for any additional thoughts or clarifications you may wish to share. Thank you once again for your thoughtful and constructive feedback.

Best regards, The Authors

We appreciate your positive comments on our work, including “novel strategy” and “interesting idea”. Detailed responses to all comments are provided below.

(1) W1 Limited contextual integration in wrap-up： Our wrap-up mechanism performs incremental semantic integration rather than relying solely on the immediately preceding segment. At each step, the text decoded from the current fMRI segment is encoded using BERT to obtain a rich semantic representation, which is then projected into the fMRI feature space of the next segment via an MLP to guide subsequent decoding. Because this process is applied recursively, the semantic embedding passed to the next segment already incorporates information from all previous segments, accumulated through the progressively decoded text. In other words, the wrap-up embedding is not merely the representation of a single sentence but a continuously integrated semantic summary that reflects the entire preceding context. We will provide a more detailed explanation of this process in the final version.

(2) W2&Q2 Unfair comparison to prior work： Following the reviewer’s suggestion, we modified UniCoRN’s decoding strategy for 60TR sequences by dividing them into 10TR segments, decoding each segment independently, and concatenating the outputs. As shown in the table below, our method still substantially outperforms UniCoRN. We attribute this superiority to two key innovations: 1) the introduction of a text-guided masking strategy during representation learning, which improves the semantic relevance and effectiveness of the learned fMRI representations; and 2) the use of a sequence-wise decoding framework with a wrap-up mechanism that leverages semantic information from preceding segments to guide subsequent decoding, thereby improving accuracy while maintaining semantic coherence in the generated text.

Additionally, the discrepancy between the results reported for UniCoRN and ours stems from a data leakage issue during its evaluation phase, which has also been reported in other paper (Yin et al., 2024). Specifically, according to the code provided by UniCoRN, its decoder model uses the same forward method for both training and inference. During inference, the decoder_input_ids (shifted ground-truth labels) are fed into the model, allowing real text to influence the generation process and thus causing data leakage. In our evaluation, we fixed this issue by removing the ground-truth input during inference. As a result, the BLEU scores decreased compared to those originally reported for UniCoRN.

[1] Yin et al., 2024. Language reconstruction with brain predictive coding from fmri data. arXiv preprint arXiv:2405.11597. 2024.

(3) W3&Q1 Text-guided masking： Our fMRI representation learning adopts a “self-supervised pretraining + text-guided fine-tuning” approach. Specifically, we first perform self-supervised pretraining on the HCP dataset (non-fMRI-text paired data) and then apply fine-tuning on fMRI-text paired data using the text masking strategy proposed in this work. The adoption of this strategy has two advantages: 1) it alleviates the limitation of scarce fMRI-text paired data; 2) it effectively guides the model to focus on fMRI time points corresponding to high-semantic-value text tokens, thereby enhancing the semantic relevance and effectiveness of the learned representations. The comparison with other representation learning methods (see Section 4.5, Table 3) validates the effectiveness and advantages of our method. Regarding the potential issue of data leakage, we clarify that the first-stage fMRI representation learning and the second-stage decoding adopt an identical data-splitting scheme. Specifically, the data are split based on stimuli into training (60%), validation (20%), and test (20%) sets, which are saved in separate directories. During both the representation learning and decoding stages, the model loads data exclusively from these corresponding directories, ensuring that the data splits used in both stages are completely consistent and that there is no overlap between the data used for pretraining and that used for decoding evaluation. Therefore, the results reported in this paper are free from any risk of data leakage.

(4) W4 Fixed Segmentation Scheme： The primary goal of this study was to validate the feasibility of the proposed brain-inspired fMRI-to-text decoding framework, particularly to examine whether the incremental decoding and wrap-up mechanisms can enhance decoding performance for long text sequences. Therefore, our focus was on evaluating the overall framework performance rather than extensively exploring segmentation strategies. To this end, we adopted a relatively simple yet effective fixed segmentation scheme and experimentally verified that a segment length of 20 TRs (approximately 60 words) achieves the best decoding performance. This finding is consistent with results from cognitive science, which show that humans achieve efficient reading and optimal comprehension with text passages of about 55 words (Dyson et al., 2001). Such consistency provides theoretical support for our choice of 20 TRs as the optimal segment length. However, as noted in both the discussion section and the reviewers’ comment, a fixed segmentation strategy has inherent limitations because it may split the narrative at unnatural points. In future work, we plan to explore content-adaptive segmentation methods that dynamically predict segment boundaries based on narrative complexity or syntactic structure, allowing the model to more flexibly adapt to diverse textual inputs.

[1] Dyson et al., 2001. The influence of reading speed and line length on the effectiveness of reading from screen. International Journal of Human-Computer Studies, 54(4): 585-612.

(5) W5 Unclear role of wrap-up in actual fMRI decoding： We conducted an additional experiment to further validate the effectiveness of the wrap-up mechanism. Specifically, we divided the 60TR fMRI sequence into three segments, performed independent decoding for each segment, and then applied GPT-2 to assess the semantic coherence between consecutive segments, and generate simple connecting phrases and adjust words in original segments to enhance consistency. As shown in the table below, our proposed wrap-up-based method achieves significantly better decoding performance compared to this post-processing approach. We will include the results of this ablation study in the final version.

(6) W6 Evaluation on Other Dataset： We conducted additional comparative experiments on a new public dataset (LeBel et al., 2023). This dataset contains fMRI signals recorded from 8 participants when they listened to 27 complete and natural narrative stories (with a total duration of approximately 6 hours). The experimental setup is consistent with that used for the Narratives dataset in the paper, and the comparison results for 60TR fMRI signals are shown in the table below. As can be seen, our method still significantly outperforms existing SOTA methods. These results will be included in the supplementary materials of our final version.

[1] LeBel et al., 2023. A natural language fMRI dataset for voxelwise encoding models. Scientific Data, 10(1): 555.

(7) Q3 Writing Error： Thank you for pointing out this writing error. We will correct it in our final vision.

Thank you for your rebuttal and addressing my comments. Below the acknowledgement and my comments:

1. Thank you for clarifying that the wrap-up embedding is updated recursively through the decoding process. This helped address my concern about the limited context integration.
2. I appreciate the effort in re-evaluating UniCoRN under a segmented setup and correcting for potential data leakage during inference. I do suggest, however, including comparisons to this work as well (Yin et al. (2024)), which you cite in your rebuttal but do not include in the main experiments. Since that work is also motivated by human language processing, a direct comparison would help position your contribution more clearly within the existing literature. Not sure why this work was not included in the paper in the first place.
3. While I understand the distinction between self-supervised pretraining and text-guided fine-tuning, I would still consider the second stage to be supervised, as it requires access to text annotations. This does not diminish the novelty of the masking strategy, but it is important to be clear about its limitations.
4. Leaving the dynamic segmentation as future work. In addition, I looked at other reviewers’ comments and the authors’ rebuttals. I agree that using the fixed setup is a natural starting point; however, I do not agree with the authors drawing strong parallels to how humans process information (e.g., rebuttal to Reviewer 4tPt and W2/Q4 mentioning "strong theoretical support" or even in the title "Human-Like Comprehension"). One can phrase that the segmented approach is inspired by how humans process information; however, I think presenting the work as building a model "like the human brain" goes way beyond what the work is about. There are many differences, for example, the use of pretraining, the wrap-up phase (where an fMRI segment is decoded into text, passed through BERT, and then the BERT representation is projected back into the fMRI latent space via an MLP to guide the next decoding step). These are all quite different from how the human brain would plausibly process information. Claiming something this strong would require much deeper investigation and a range of analyses to support it. The work itself remains firmly on the application side, namely focused on advancing state of the art in fMRI-to-text decoding. Referencing findings like "using 60 words is consistent with 55-word reading chunks that support efficient comprehension" (Dyson et al., 2001) is an interesting observation, but it does not come close to justifying a claim that the method is decoding language like the human brain. The architecture is entirely different, and there is no analysis connecting model behavior to actual brain activity.
   • To be clear, I see value in the paper, especially in its contributions to improving decoding performance. But extending that into claims about brain-like processing is simply not supported. In fact, thinking about it now, it feels excessive to have that statement even in the title. Until reading the rebuttal, I had treated the cognitive framing as light motivational context and not as a core part of the novelty.

5 & 6) Including the additional experiments for better understanding on the role of wrap-up phase, and generalization of the method to new datasets.

I am willing to potentially increase my score by 1 in light of the additional experiments and clarifications provided. However, it’s important to be cautious not to overclaim novelty by drawing strong parallels to human processing. In my view, this work does not offer any explanation or insight into how the brain actually processes language, nor does it establish a meaningful connection between the proposed model and human cognitive function. The parallels drawn, such as the 55- vs. 60-word segmentation, are, at best, curious observations that would require much deeper investigation to support any stronger interpretation. Ultimately, this is an application-focused paper: it presents a well-designed method and shows how computational modeling can push the state of the art in fMRI-to-text decoding. In my opinion, that is where its strength lies, not in modeling or explaining brain function.

We sincerely appreciate the reviewers’ positive feedback on our research topic, experimental results, and overall presentation of the paper. Below, we provide detailed responses to all comments.

(1) W1 Novelty of Work： Our work is very novel on achieving better brain signal decoding, in particular, addressing or mitigating the insufficiency of data and long-context challenges. First, we propose a brain-inspired sequential decoding paradigm for fMRI-to-text decoding that incorporates both incremental processing and segmental integration. The motivation for this framework arises from the fundamental differences between fMRI-to-text decoding and conventional cross-modal translation tasks. Unlike translation, which performs a one-to-one mapping between two independent modalities, fMRI-to-text decoding reconstructs textual content from neural activity patterns generated during language comprehension. As a result, models designed for cross-modal translation are not directly applicable to this task. Based on this insight, we hypothesize that adopting a decoding framework more aligned with human language processing mechanisms could better address this challenge. Our experimental results (Section 4.3, Table 1) and ablation studies (Section 4.4, Table 2) support this hypothesis, demonstrating the feasibility of brain-inspired fMRI-to-text decoding and providing a new perspective for future model design. In the final version, we will add a clearer explanation of the motivation behind this brain-inspired framework. Second, we design a text-guided masking strategy during fMRI representation learning to guide the model to focus on fMRI time points that correspond to high-value text tokens, thereby enhancing the semantic relevance and expressiveness of the learned fMRI representations. The comparison with other representation learning methods (Section 4.5, Table 3) further demonstrates the effectiveness and advantages of this approach.

(2) W2&Q1 Fixed Segmentation Scheme： The fixed segment length of 20 TRs was determined via systematic validation in our work (Section 4.2, Figure 4). This finding is roughly consistent with evidence from cognitive science, which shows that text passages of approximately 55 words support efficient reading at both normal and fast speeds and lead to the highest comprehension levels (Dyson et al., 2001). Given that each TR (1.5s) corresponds to roughly three words, a 20TR segment covers approximately 60 words, aligning well with the optimal text length for comprehension reported in. This provides theoretical support for selecting 20 TRs as the optimal text segment length in our work. Since the main objective of this study was to validate the feasibility of applying brain-inspired mechanism to fMRI-to-text decoding, our focus was on evaluating the overall framework performance rather than extensively exploring text segmentation strategies. As discussed in our paper and highlighted by the reviewers, our future work will further investigate content-adaptive segmentation methods that dynamically predict segment boundaries based on narrative complexity, enabling the model to more flexibly adapt to diverse textual inputs.

[1] Dyson et al., 2001. The influence of reading speed and line length on the effectiveness of reading from screen. International Journal of Human-Computer Studies, 54(4): 585-612.

(3) W3&Q3 Examples of Decoded Text： Due to page limitations, detailed examples of decoded text are provided in the supplementary materials. Specifically, in Section A.3 ("Text Cases") of the Appendix, we provide representative decoded examples for our method and SOTA approaches across different input lengths. Our proposed method outperforms SOTA methods in terms of capturing semantics and syntax in tokens and captures more key content words ranging from verbs to nouns, including more accurate named entities such as person and place names, and produce sentences that are semantically more aligned with the intended meaning. Therefore, our method produces decoded outputs that exhibit higher word-level overlap and better semantic alignment with the reference texts, demonstrating superior decoding performance. In the final version, we will include some of the decoded text in the main body of the paper.

(4) Q2 Novelty of wrap-up mechanism： In our framework, the proposed wrap-up mechanism is superficially similar to existing state-passing mechanisms but differs fundamentally in both the modality and the structural flow of the transferred information. Traditional state-passing mechanisms typically operate within a single modality (e.g., text-to-text), propagating hidden states from one segment to the next to maintain temporal continuity, akin to how recurrent neural networks or autoregressive models carry over internal states across time steps. In contrast, our wrap-up mechanism is a cross-modal semantic summarization process: it decodes an fMRI segment into text, encodes the generated text using BERT to extract a rich semantic representation, and then projects this representation back into the fMRI latent space of the subsequent segment via an MLP to guide future decoding. This is not merely state propagation, but a semantic-level integration that involves modality translation and higher-level abstraction. While our method shares superficial resemblance to state-passing, particularly in its sequential setup, it is structurally distinct. State-passing emphasizes retention and transfer of temporal internal states (e.g., hidden activations) across steps, whereas our wrap-up mechanism summarizes each fMRI window into a semantic embedding that represents integrated content, without explicitly transferring low-level hidden states. In that sense, it is more analogous to a block-wise summarization approach, such as using a [CLS] token or attention-based pooling to encode the essence of an input segment. We believe this difference is not just architectural but also functional: the wrap-up mechanism aims to provide a semantic bridge across fMRI segments by leveraging language-domain priors, rather than preserving low-level continuity.

Dear Reviewer 4jo4,

We sincerely appreciate the time and effort you have devoted to reviewing our submission. We have carefully prepared our rebuttal, aiming to address the valuable points you raised in a direct and respectful manner.

Should you have an opportunity to review our response, we would be grateful for any additional thoughts or clarifications you may wish to share. Thank you once again for your thoughtful and constructive feedback.

Best regards, The Authors