MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-text Decoding

C1 It is unclear how the fMRI encoder handles different numbers of voxels.

R1 Each voxel is treated as a token in our model, and the attention layer learns to map sequences of varying lengths into a fixed-dimensional representation. This is similar to using multiple [CLS] tokens in a BERT model to capture the semantics of sentences of various lengths. This works because, in attention, the output dimension is solely determined by the number of queries. We also provide a mathematical clarification

C2 The idea is not entirely novel—see MindEye2.

R2 We respectfully disagree with the statement and would like to emphasize the fundamental differences between our approach and MindEye2:

1. MindEye2 requires a separate projector layer trained on each subject to map their features into a shared representation space.
2. The architecture of MindEye2 cannot handle subjects with varying numbers of voxels.

Therefore, MindEye2 cannot generalize to unseen subjects without further tuning on data from the unseen subject. In contrast, our model does not need subject-specific parameters or architectures and can generalize to held-out subjects in a zero-shot manner.

C3 The brain instruction tuning lacks details of backpropagation.

R3 The brain instruction tuning trains the model through the loss function in lines 254-255. The gradients are backpropagated from the predicted tokens to the LLM, and all the way back to the fMRI encoder to update its parameter.

C4 No comparison to show the results w/ and w/o the brain instruction tuning.

R4 We would like to politely point out that we did have a comprehensive experiment to show the effects of brain instruction tuning (BIT). As shown in Table 2’s caption, models with $\circ$ are the versions w/o BIT. We can see models w/ BIT significantly outperform their corresponding models w/o BIT. On average, BIT brings 28.0% improvement, as stated in line 340, right column.

C5 Figure 6 is difficult to interpret.

R5 We improved Figure 6's clarity by selecting a more perceptually friendly colormap. We also improved the caption to better clarify that each subfigure corresponds to a query token on subject 1. The findings are discussed in section 4.7.

C6 Brain maps across subjects should be included.

R6 We thank the reviewer for the suggestion. We now present the query token from Figure 6(a) applied to additional subjects (Subjects 2–7) below.

Figure

We observe that the attention maps exhibit highly similar spatial patterns across subjects. This consistency supports the generalizability across subjects to capture common brain patterns.

C7 The shared vs. subject-specific information in the latent space should be verified.

R7 We do not encourage the encoding of subject-specific information in the method, as our model is designed to be subject-agnostic. This enables scalable deployment across diverse populations and eliminates the need for costly, subject-specific calibration.

To assess how subject-specific and subject-agnostic information evolve through the model, we visualize the latent embeddings at various stages of the encoder below.

Figure

C8 Performances on individual subjects should be included.

R8 We appreciate the suggestion. The results in Table 2 are based on subject 1. We have now extended our experiments to include subjects 2 and 5 below.

Table

Across subjects, our model outperforms the baselines in most cases, demonstrating robustness to inter-subject variability. We note that performance varies slightly more on ST-VQA and TallyQA. Models trained on subject 5 outperform those trained on subjects 1 and 2, suggesting that subject 5 provides higher-quality data.

We will include results for all subjects in the revised version.

C9 What are the key conclusions from MindLLM? The paper lacks a discussion of the findings and implications of the current work.

R9 While interpretation (Figure 6) supports our model’s motivation, our primary focus is improving decoding performance.

Key findings:

• Brain instruction tuning significantly improves the performance.
• Neuroscience-informed attention significantly outperforms vanilla cross-attention (Section 4.6), offering architectural insights for fMRI decoding.

Impact:

Our method enables subject-agnostic decoding and easy task adaptation, which unlocks out-of-the-box applications like prosthetic control without subject-specific finetuning. We will update detailed discussions in the revised version.

C1

The manuscript lacks comparisons or references on brain-machine alignment or image/video decoding [1][2][3].

R1

We thank the reviewer for pointing out these relevant references. [1] [2] deal with fMRI time series, while our method focuses on static fMRI (i.e., fMRI signals at a moment). Therefore, they are not comparable to our work. We will create a new section for multimodal brain decoding in related work and discuss these models in the revised version. For [3], we have included it in the baselines. The results are shown below and will be updated in the revised version (The MindVis column is the new baseline).

Table

C2

The framework is not very novel, as it directly mimics the visual instruction tuning.

R2

The training paradigm of brain instruction tuning is similar to visual instruction tuning. However, the construction of the brain instruction tuning datasets is a nontrivial contribution. As discussed in section 3.3 (lines 177-184, right column), we select datasets to capture diverse aspects of semantic information embedded in fMRI signals, which are considered among the most fundamental and essential properties of human brains.

Q1 The computational cost for model's training and inference, compared to other baselines.

A1

We thank the reviewer for this valuable question. We summarize the computational complexity and runtime analysis below.

Table

Experiments here were conducted on run on an A40. All timings are measured during inference with batch_size=1. When comparing with other encoders, we only measure the encoder part (LLM is excluded). We separately time the LLM at the last row.

Despite its superior performance, our model introduces only a marginal increase in encoder-side computational cost compared to other baselines. Notably, the majority of inference time and complexity arises from the LLM component, which is shared across all models. This highlights that the design choices in the encoder—while crucial for performance—do not significantly affect runtime.

Q2

Does scaling laws of data observed in the results, more subjects trained will increase the accuracy?

A2

We appreciate the reviewer’s valuable question and suggestion. We conducted experiments to evaluate how model performance scales with the number of subjects and report the performances on the COCO caption task. We examined both in-distribution (seen subjects) and out-of-distribution (held-out subjects) settings.

Table

Our show significant performance improvements as the number of training subjects increases, demonstrating that the model benefits from exposure to more subjects. We will include more results on other subjects and datasets in the revised version.

Q3

How is subject-specific information encoded in this framework?

A3

We do not encourage the encoding of subject-specific information in the method, as our model is designed to be subject-agnostic—capable of generalizing to individuals not seen during training without requiring additional fine-tuning. This "out-of-the-box" generalization has significant practical benefits: it supports scalable deployment across diverse populations and eliminates the need for costly, subject-specific personalization or calibration.

To assess how subject-specific and subject-agnostic information evolve through the model, we visualize the latent embeddings at various stages of the encoder using T-SNE below.

Figure

This figure illustrates the transition from subject-specific to subject-agnostic representations. Initially, the embeddings exhibit distinct subject-wise clusters, indicating strong subject-specific information. After passing through the neuroscience-informed attention layer, these clusters are still visible. However, as the embeddings propagate through successive MLP layers, the subject-specific patterns gradually dissolve. By the final layer, the latent space forms a well-mixed representation where subject identity is no longer distinguishable.

This transformation is crucial for enabling generalization to unseen subjects in a zero-shot manner.

C1

The paper claims "a voxel’s position alone can theoretically serve as effective keys for attention weight computation", but no evidence is provided.

R1

We would like to point out politely that it is not a guess—it is strongly supported by the ablation study (note the blue line) in section 4.6. As pointed out in lines 403-406, left column,

The vanilla cross attention (Pos Enc. + fMRI) leads to poor performance, while removing fMRI values from the key embeddings (Pos Enc.) yields a significant improvement.

which validates the hypothesis of using a voxel’s position alone.

C2

For Tables 1, 2 & 3, It is not clear how much of the improvement is due to the proposed model versus the use of superior LLMs. It is unfair to compare their performances with other state-of-the-art baselines that do not use the same LLM in Table 1.

R2

We argue that the comparison is fair across Table 1-3, as pointed out in lines 140-142

In practice, we use Vicuna-7b (Zheng et al., 2023) as our LLM to maintain consistency with our baseline (Xia et al., 2024).

Specifically,

• Table 1: The strongest baseline, UMBRAE, uses Shikra-7b, a vision-language model improved based on Vicuna-7b—consistent with our choice of LLM.
• Table 2: Besides UMBRAE, for other competitive baselines (MindBridge, UniBrain), we use the same LLM backbone as ours.
• Table 3: All baselines share the same LLM backbone as our model.

Furthermore, as discussed in Section 4.2, we observe substantial performance gains attributable to the brain instruction tuning and the encoder design. This supports our claim that the improvements stem from our proposed approach rather than solely from the choice of LLM.

C3

The results for METEOR, CIDEr and SPICE of the proposed method are only slightly superior to or even inferior to the baselines.

R3

While our model does not significantly outperform the baselines in terms of METEOR, CIDEr, and SPICE, it offers several key advantages that the baselines lack:

• As shown in Table 1, our model is the only one that is subject-agnostic. In other words, it can generalize to unseen subjects in a zero-shot manner. This is particularly valuable in BCI applications, where users often expect the device to work out of the box without requiring subject-specific training data.
• As noted in lines 82–83 (right column), our model can handle tasks beyond simple image-stimulus associations—such as answering memory-related questions—while the strongest baseline, UMBRAE, cannot.

We would also like to emphasize that our method demonstrates consistent and broad improvements across a wide range of downstream tasks. For example, beyond brain captioning, our model outperforms all baselines on 33 out of 38 evaluation metrics reported in Table 2.

C4

The ablation study is not sufficiently convincing. There is no explanation of the task being experimented on, and the authors should provide the final results, not just the loss.

R4

We appreciate the reviewer’s feedback and apologize for the lack of clarity regarding the experimental setup. The ablation study is conducted across all tasks during brain instruction tuning on subject 1, and the reported loss is the average loss across these tasks. We will update the details about settings in the revised version.

Our decision to present loss curves was intentional: As pointed out in lines 405–411, left column, the comparison of the convergence speeds of the orange and green lines helps illustrate the impact of both region and positional encodings. (We do not plot metrics during training at each step since generation is time-consuming)

However, we agree that final performance metrics are essential for a complete picture. We have now computed these metrics for most downstream tasks, and the results are shown below. The results are consistent with our original claim based on the loss curves. Note that it also validates our response to the reviewer's concern 1.

Table

We will include the final results of all tasks in the revised version.

C1

The text claims UniBrain is subject-agnostic, but Table 1 lists MindLLM as the only subject-agnostic model—this inconsistency needs clarification.

R1

We apologize for the mistake and thank the reviewer for bringing this to our attention.

with only a few exceptions (Mai & Zhang, 2023).

Should actually be

with only a few exceptions (Wang et al., 2024b)

The models in the two references share the same name, which led to the citation error. We will correct it in the revised version.

C2

Line 32 "results" should be upper-case at the beginning of the sentence. Something is also off about the grammaticality of the sentence.

R2

We thank the reviewer for pointing this out. We have corrected the capitalization and revised the sentence for clarity and grammatical correctness. The updated version reads:

The results demonstrate that our model outperforms the baselines in downstream tasks, generalization to unseen subjects, and adaptation to novel tasks.

Q1

How well does your model scale with the number of subjects? I.e. how much better does it perform with pre-training on e.g. 3 subjects compared to 1 etc?

A1

We appreciate the reviewer’s valuable question and suggestion. We conducted experiments to evaluate how model performance scales with the number of subjects and report the performances on the COCO caption task. We examined both in-distribution (seen subjects) and out-of-distribution (held-out subjects) settings.

Table

Our results show significant performance improvements as the number of training subjects increases, demonstrating that the model benefits from exposure to more subjects during pre-training. We will include more results on other subjects and datasets in the revised manuscript.