Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex

We appreciate your questions and concrete suggestions! We respond to specific questions below. We look forward to further discussion, and welcome additional comments.

Q1) Stochastic visual response functions

You are correct. Our current framework assumes a true response function that is noiseless, and is only affected by the current stimulus. Due to this assumption, our encoder outputs are also deterministic in nature. We adopted this setup because these conditions are commonly (sometimes implicitly) assumed in other fMRI encoder works.

If we want to construct an encoder that can capture the inherent distribution of a stochastic visual response function, we need to adopt a stochastic generative framework. There has been recent work on adopting diffusion models as fMRI encoders [1]; however their work currently does not capture the trial variability due to averaging across stimulus repeats. In such a framework, the variability across subjects, machines, stimuli, repeats, etc. can be modeled by introducing additional conditioning variables to the generative framework.

We agree this is an interesting future direction, and stochastic fMRI encoders could allow us to more accurately model the visual response.

Q2) Differentiability of the visual response function

To clarify, we do not assume that the true biological visual response function is inherently differentiable. Rather, we employ differentiable functions as computational approximations to model these unknown response patterns. We utilize differentiable functions because they can be optimized using modern gradient-based optimization techniques popularized by deep learning.

Modern neural encoding models in computational neuroscience commonly adopt this approach (such as [1,2]) because differentiable neural networks provide universal approximation capabilities, meaning they can approximate non-differentiable functions to high precision [3,4].

As shown in our Experiments section, BraInCoRL demonstrates that this differentiability constraint does not meaningfully limit our model's ability to capture biologically relevant response patterns. The successful identification of known category-selective regions, strong cross-dataset generalization, and attention patterns all indicate that differentiable approximations effectively capture the essential semantic selectivity of higher visual cortex.

Q3) Choices of Losses

This is a good question! Per your suggestion, we conduct an ablation study on the choice of different loss functions used to optimize the network on real neural data.

Our experimental results show that MSE and L1 losses achieve similar performance across all context sizes, with minimal differences. This suggests that both metrics are equally effective for capturing voxelwise neural response patterns. We also explore the hybrid loss of 0.5 × MSE loss + 0.5 × (1 - cosine similarity), where the cosine loss is applied on ground truth voxel weights that are derived from ridge regression. We find this underperforms by 2-4% compared to MSE/L1. We believe this is due to the higher weighting on directional correctness of the weights, rather than optimizing for prediction error minimization.

We are motivated to adopt MSE since it corresponds to OLS regression commonly used for fMRI. MSE loss is also suitable because it provides optimal performance.

• Voxel-wise explained variance of different training losses with the CLIP backbone for Subject 1 (higher is better).

Q4) Clarity of text

Thank you for pointing out the typos and labels in our paper! We will update the paper according to your suggestions and those from other reviewers to improve the clarity of our upcoming revision.

Conclusion

We are grateful for your suggestions and detailed comments. We will incorporate your feedback and our corrections into the upcoming revision. We look forward to discussing with you further!

[1] MindSimulator: Exploring Brain Concept Localization via Synthetic FMRI (2025).

[2] A differentiable approach to multi-scale brain modeling (2024).

[3] Multilayer feedforward networks are universal approximators (1989).

[4] Approximation capabilities of multilayer feedforward networks (1991).

We appreciate the concrete feedback and comments! We address your suggestions below.

Q1) Performance gain from real neural data

This is a good question. To clarify, Figure 3b evaluates performance at a fixed in-context support size of 100 images on a held out subject. To further evaluate the full impact of real-data finetuning, we visualize the trends as shown in Section A.6 on page 30 of the Supplement. We find that finetuning yields substantial performance gains -- over 20% improvement in variance explained across support set sizes compared to the pretrained model. This effect becomes especially pronounced at larger context sizes (>500), where the synthetic pretrained model often falls below the Ridge regression baseline, while the fully trained BraInCoRL model continues to improve.

Beyond raw performance, we want to highlight how BraInCoRL leverages the structure of neural data. During Stage 3 (real-data finetuning), the model meta-learns across a distribution of voxel-level tasks -- each defined by a voxel’s stimulus-response mapping in a specific subject. BraInCoRL uses a transformer-based hypernetwork conditioned on in-context stimulus-response pairs. This design allows the model to capture shared representational structure across subjects and voxels, and adapt to novel subjects and voxels through in-context learning, without test-time training.

Q2) Finer-grained selectivity

We agree that current models (our model, as well as those from other researchers) based on linear projections of features from a frozen foundation model vision transformer, impose strong constraints on the final fMRI encoding model. This may make the learning of fine-grained features that drive voxel activity difficult. We would like to clarify that our reported explained variance is not normalized to the effective noise ceiling -- and the explained variance of our models are very competitive with recent state-of-the-art work [1] that utilize a vision transformer as the image backbone.

In the future, a promising direction for fMRI encoders may be constructing non-linear mappings between backbone features and the brain, or those that can capture a distribution of activations based on stochastic generative models. We anticipate that a primary driver of this type of work will be the availability of higher quality and more diverse fMRI datasets.

Q3) Model architecture

The architecture of our model is composed of:

(1) An input projection MLP layer is applied to each token individually, which maps the stimulus semantics and voxel activation into an embedding.

(2) 20 Transformer encoder layers that integrate information across all in-context examples and computes trends for voxel activation across multiple stimuli.

(3) The [CLS] token from the final layer goes through an MLP to output a hyperweight which is used to parameterize the final encoder.

Our code is provided on line 22 of the main paper. We will update the paper revision to include additional model details.

Q4) Typo clarification

In line 131, we correct the context token $c_i$ to be $=[x_i; \beta_i]$, so $\beta$ will be consistent in the previous BOLD signal representation in line 129.

Q5) Logit scaling equation clarification

Here we clarify the use of logit-scaling in Equation (4) for our model.

Logit scaling is motivated by the need for robust generalization across varying context sizes. Specifically, it helps the in-context encoder maintain stable performance regardless of the number of conditioning stimuli, including those beyond the training range. Although we train with between $30$ and $500$ images, practical applications may involve fewer than $30$ or more than $500$. Logit scaling ensures the model remains functional and accurate even in these extrapolated settings.

This logit scaling approach was originally proposed in [2, 3], and later adopted by Qwen LLM (logn-scaling) [4] & Llama 4 LLM (temperature scaling) [5]. Below, we provide a brief summary of the high-level mathematical rationale based on [2], along with our own commentary:

The self-attention value for query token $i$ to value token $j$ is

$a\_{i,j} = \frac{e^{\lambda \mathbf{q}\_i \cdot \mathbf{k}\_j}}{\sum\_{j=1}^n e^{\lambda \mathbf{q}\_i \cdot \mathbf{k}\_j}}$

Then we define the entropy of query token $i$ as $\mathcal{H}\_i = -\sum\_{j=1}^n a\_{i,j} \log a\_{i,j}$

If we substitute the expression for $a\_{i,j}$, we will have

$\mathcal{H}\_i = -\sum\_{j=1}^n a\_{i,j} \log \left(\frac{e^{\lambda \mathbf{q}\_i \cdot \mathbf{k}\_j}}{\sum\_{j=1}^n e^{\lambda \mathbf{q}\_i \cdot \mathbf{k}\_j}}\right)$ Since $\sum\_{j=1}^n a\_{i,j} = 1$,

we can express the entropy as

$\mathcal{H}\_i = \log \sum\_{j=1}^n e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j} - \frac{\sum\limits\_{j=1}^n e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}(\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j)}{\sum\limits\_{j=1}^n e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}}$

If the first term is expressed as an expectation over $j$, we have

$\sum\_{j=1}^n e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j} = n\times \frac{1}{n}\sum\_{j=1}^n e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}\approx n\mathbb{E}\_j[e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}]$

Then we can derive the following approximation of entropy $\mathcal{H}\_i \approx \log n + \log \mathbb{E}\_j[e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}] - \frac{\lambda\mathbb{E}\_j[e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}(\mathbf{q}\_i\cdot \mathbf{k}\_j)]}{\mathbb{E}\_j[e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}]}$

Assuming that the vectors are the output of a layernorm layer with length $\sqrt{d}$, the expectation can be converted to one over the angles between vectors

$\mathcal{H}\_i \approx \log n + \log \mathbb{E}\_{\theta}[e^{\lambda d \cos\theta}] - \frac{\lambda d\,\mathbb{E}\_{\theta}[e^{\lambda d \cos\theta}\cos\theta]}{\mathbb{E}\_{\theta}[e^{\lambda d \cos\theta}]}$

Since randomly distributed vectors in high-dimensional spaces are mostly nearly orthogonal, we derive a term that can be approximately expressed as

$\mathcal{H}\_i \approx \log n + C$ where $C$ does not depend on the number of tokens $n$.

This yields an approximate scaling factor for the logits of $\log n$, which helps maintain the entropy of the attention distribution across different context lengths.

As a result, we apply logit scaling with a factor of $\log n$ to the $QK^T$ term in the attention mechanism, changing the standard attention value formulation.

Note that in the explanation we adopt the notation from [2].

We have provided a more detailed discussion of the use of Equation (4) in the revised version of our paper.

Q6) BraInCoRL conditioned on 9000 images

This is a great suggestion. We have added an additional experiment to evaluate the performance of BraInCoRL with 9000 in-context samples using the CLIP backbone, for the four NSD subjects (S1, S2, S5, S7) in our main paper. Note that in this case, the model has not seen the subject we test on during training, and the model was trained on the other three subjects.

As shown in the table, the performance difference is less than $0.01$ explained variance difference across all subjects when compared to a fully trained model (which is fit on each subject's entire 9000-image training set). This means BraInCoRL achieves 94-99% of the fully trained model's performance. Crucially, BraInCoRL reaches this near-optimal performance while requiring no parameter updates or gradient-based optimization during the in-context inference.

• Voxel-wise explained variance of BraInCoRL with the CLIP backbone compared with the fully trained reference model. The difference variance explained is less than 1% (higher is better).

Q7) Number of images clarification

In the result on page 6, we intend to say that we can utilize up to each subject's 9000 unique images in the NSD dataset as the in-context support set. But during the evaluation experiment (such as shown in Table 1 caption), we just randomly sample a subset of e.g. 100 in-context images as the inference input. We have clarified this in the revision.

Q8) Size of BOLD5000

This discrepancy arises from the inherent nature of the BOLD5000 dataset itself. The NSD dataset involves up to $10,000$ images for each subject being viewed up to three times. In contrast, for BOLD5000, only approximately $100$ images were viewed more than three times.

We were advised by the BOLD5000 authors to only analyze the image subset with more than three repeats. Following our discussions with the BOLD5000 dataset authors, we follow the sampling strategy below: we randomly sample 20% of the BOLD5000 images as inference images, with the remaining 80% used as the in-context support set.

Conclusion

We are very grateful for the highly detailed and actionable suggestions. We have included additional details for our method, and conducted additional experiments. We look forward to any additional questions.

[1] Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset (2023).

[2] Analyzing the scale operation of attention from the perspective of entropy invariance (2021). (see line 564 of main paper).

[3] Overcoming a Theoretical Limitation of Self-Attention (2022).

[4] Qwen technical report (2023).

[5] The Llama 4 herd: The beginning of a new era of natively multimodal ai innovation (2025).

This is an interesting question. We agree this is a possibility.

To expand on this, our work and ridge regression utilize different inductive biases. Ridge regression penalizes strong contributions from individual features and is equivalent to a zero-mean gaussian prior with variance determined by the penalty term. On the other-hand, our framework learns a data-driven inductive bias over many voxel/stimuli pairings. As stated by the "No Free Lunch" theorem, there is no universally better learning algorithm -- so there will be cases where ridge may outperform our method when the visual response falls significantly outside our training distribution.

However, fMRI's inherent spatial structure favors our approach. Each voxel aggregates signals from ~100,000 neurons, inherently smoothing population-level neural responses. This spatial integration:

1. Suppresses single-neuron variability
2. Creates low-dimensional representations of neural population activity (think of the extreme case where a voxel is an average of all neurons in the brain)
3. Creates regularities that data-driven methods can utilize

Ridge regression’s static bias (weight shrinkage) cannot explicitly leverage this structure, while our data-driven method can.

We appreciate your excellent suggestions! We will incorporate all of your feedback into our paper.

Q1) Training cost and complexity

We agree with you that the training stage of our model is more costly than traditional fMRI encoders. However this cost gains us significant generalization ability.

Our current model already achieves strong generalization performance on fMRI data that was collected using a different scanner, voxel size, experiment protocol, and human subjects -- without requiring any finetuning or training on these smaller datasets. We also achieve significant data-efficiency gains, demonstrating better performance in the low data regime compared to traditional baselines.

Due to this, our training & inference code and weights will be made available under a permissive open source license. Future researchers will be able to adopt our model for their datasets without training from scratch (and so will not be limited to their small datasets), and benefit from the models that we have already trained.

Q2) Contribution of each stage

The three-stage training is essential, with each stage addressing a distinct challenge.

• 1st stage: Synthetic Pretraining enables comprehensive coverage of the visual response space. It provides the model with an initial and broad feature representation by exposing it to a large volume of synthetically generated data.
• 2nd stage: Context Extension trains the model on variable-length contexts, enabling robust length generalization, and allows for test-time context size to be outside of the range seen during training.
• 3rd stage: Real-Data Finetuning adapts the model to the statistical properties of real fMRI signals, in order to bridge the gap between synthetic and biological domains. Note that in the main paper, for evaluation fairness we avoid training on the same subject that we evaluate on.

This progression from synthetic foundation → context flexibility → biological adaptation ensures that each fundamental challenge—response function coverage, variable context handling, and biological realism—is systematically addressed in the optimal order.

Due to OpenReview limitations, here we present results only for the first two subjects of the NSD dataset; however, the observed trends and performance are consistent across all other subjects.

• Voxel-wise explained variance for BraInCoRL with CLIP backbone for NSD Subject 1 on different training stages, compared with ridge baseline and fully trained reference model (higher is better).

• Voxel-wise explained variance for BraInCoRL with CLIP backbone for NSD Subject 2 on different training stages, compared with ridge baseline and fully trained reference model (higher is better).

In stage 1, we find that training with synthetic data without context extension limits the performance at higher context sizes. After stage 2, the model gain the ability to consistently improve performance as context size increases. In stage 3, after being exposed to real neural data, our model further improves in performance.

Conclusion

We thank you for your comments, and we hope that this clarifies our results! We have further clarified the contributions of each training stage, and will update the paper to reflect your suggestions.

Hello,

We hope our response has been helpful! We would be happy to discuss further.

As a gentle reminder, due to NeurIPS requirements this year -- you may have to submit a "Mandatory Acknowledgement" and edit your review with a "Final Justification".

Thank you!

We appreciate your very thoughtful and helpful suggestions. Below are our responses. We look forward to further discussion, and welcome additional comments.

Q1) Notation regarding token indexing

The context tokens in line 129 and line 132 should be indexed by the same variable and extend to the same count. This is a typo that we have updated in our upcoming revision. In both cases, the number of context tokens during inference represents the number of stimuli that a subject has seen -- which are then made available to the model for in-context learning.

We will update this in the revised paper.

Q2) Notation regarding voxel signal

We agree that the clarity here could be improved. Per your suggestion, we have unified the notation around voxel $v$ and beta value $\beta$. Our original intention was to use $\beta$ since the voxel's response to a given image in fMRI viewing is typically derived as the beta coefficients of a GLM regression. We intend the two variables to represent the same value, which is a scalar representing a voxel's response to a given stimulus.

This has been changed in our revision.

Q3) Principles underlying logit scaling

The motivating factor underlying logit scaling is our desire to have our in-context learned encoder perform well regardless of the number of stimuli given to the model, and effectively generalize to context sizes beyond those seen during training. For example, while we may only train with between $30$ and $500$ images, a third-party experimenter may want to use fewer than $30$ images or more than $500$ images to condition the model. Across all cases, we want the model to succeed.

This logit scaling method was first proposed in [1, 2], and later adapted in Qwen LLM (logn-scaling) [3] & Llama 4 LLM (temperature scaling) [4]. We will briefly summarize the high-level math, which we take from [1] with our commentary:

Let the attention value in self-attention for a particular query token $i$ to value token $j$ to be

$a\_{i,j} = \frac{e^{\lambda \mathbf{q}\_i \cdot \mathbf{k}\_j}}{\sum\_{j=1}^n e^{\lambda \mathbf{q}\_i \cdot \mathbf{k}\_j}}$

Then the entropy is defined as $\mathcal{H}\_i = -\sum\_{j=1}^n a\_{i,j} \log a\_{i,j}$

Subsituting the expression for $a\_{i,j}$ we have the entropy as

$\mathcal{H}\_i = -\sum\_{j=1}^n a\_{i,j} \log \left(\frac{e^{\lambda \mathbf{q}\_i \cdot \mathbf{k}\_j}}{\sum\_{j=1}^n e^{\lambda \mathbf{q}\_i \cdot \mathbf{k}\_j}}\right)$ Since $\sum\_{j=1}^n a\_{i,j} = 1$,

we can express the formula as

$\mathcal{H}\_i = \log \sum\_{j=1}^n e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j} - \frac{\sum\limits\_{j=1}^n e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}(\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j)}{\sum\limits\_{j=1}^n e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}}$

If the first term is expressed as an expectation over $j$, we have

$\sum\_{j=1}^n e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j} = n\times \frac{1}{n}\sum\_{j=1}^n e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}\approx n\mathbb{E}\_j[e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}]$

Which leads to the following approximation of entropy

$\mathcal{H}\_i \approx \log n + \log \mathbb{E}\_j[e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}] - \frac{\lambda\mathbb{E}\_j[e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}(\mathbf{q}\_i\cdot \mathbf{k}\_j)]}{\mathbb{E}\_j[e^{\lambda \mathbf{q}\_i\cdot \mathbf{k}\_j}]}$

If the vectors are assumed to be the output of a layernorm layer with length $\sqrt{d}$, the expectation can be converted to one over the angles between vectors

$\mathcal{H}\_i \approx \log n + \log \mathbb{E}\_{\theta}[e^{\lambda d \cos\theta}] - \frac{\lambda d\mathbb{E}\_{\theta}[e^{\lambda d \cos\theta}\cos\theta]}{\mathbb{E}\_{\theta}[e^{\lambda d \cos\theta}]}$

Since most randomly distributed vectors in higher dimensions are orthogonal, we derive a term which can roughly be expressed as

$\mathcal{H}\_i \approx \log n + C$ where $C$ does not depend on the number of tokens $n$.

This leads to an approximate scaling factor for the logits of $\log n$ to keep the entropy invariant to context length.

Therefore, we change the standard formulation of attention values by applying a scaling factor of $\log n$ to the $QK^T$ term, as shown in Equation (4) of our paper.

Note that in the above explanation we adopt the notation from [1].

We have provided a more detailed discussion of the use of logit scaling in the revised version of our paper.

Q4) Why explained variance?

Explained Variance is a performance measuring metric that quantifies the proportion of the total variance in the fMRI signal that can be predicted by our BraInCoRL encoding model.

We follow prior work that utilizes the NSD fMRI dataset, which primarily evaluate Explained Variance as the metric of encoding model performance [5, 6, 7] -- note all three of these papers have one NSD author as co-author.

We agree that Pearson $R$ is also a possible metric we can use to evaluate encoder performance. In a sense, Pearson $R$ is more forgiving metric. Pearson $R$ is affected neither by scaling, nor by constant bias, while explained variance is only robust to bias. The use of Pearson $R$ reflects an implicit preference for the model to capture the right trend in predicted responses while caring less about capturing the exact value.

Note that BOLD5000 contains distribution shifts for several reasons - for example, it incorporates COCO, ImageNet, and SUN images. With NSD, we explicitly controlled the training and evaluation responses distributions to be the same, making explained variance appropriate as a strong criterion. In contrast, with BOLD5000, even though every voxel was still z-scored, there are still many factors that are different compared to NSD (MRI scanner quality, more variable task). After discussion with BOLD5000 authors, we were advised to evaluate Pearson $R$ for BOLD5000, rather than the more stringent explained variance (see also the GLMSingle paper for a direct comparison of the datasets).

Q5) Other metrics for NSD

Here we evaluate Pearson $R$, and Spearman's rank correlation coefficient (Spearman's $\rho$) for NSD dataset. Note that in this case, the BraInCoRL model has not been trained or finetuned on Subject 1, while the Fully Trained model is trained on $9000$ images from this subject. Our model can approach within 10% of the Fully Trained model using just $300$ images when evaluated using Pearson $R$.

• Voxel-wise Pearson R for BraInCoRL, within-subject ridge regression baseline and fully-trained reference model (NSD dataset, CLIP backbone, Subject 1, higher is better).

• Voxel-wise Spearman's $\rho$ for BraInCoRL, within-subject ridge regression baseline and fully-trained reference model (NSD dataset, CLIP backbone, Subject 1, higher is better).

Q6) Other metrics for BOLD5000

Here we evaluate explained variance and Spearman's rank correlation (Spearman's $\rho$) for BOLD5000. Our BraInCoRL model was trained on NSD, while the Ridge Baseline was fit on BOLD5000. We achieve significantly better performance.

• Voxel-wise EV for BraInCoRL and within-subject ridge regression baseline (BOLD5000 dataset, CLIP backbone, Subject CSI1, higher is better).

• Voxel-wise Spearman's $\rho$ for BraInCoRL and within-subject ridge regression baseline (BOLD5000 dataset, CLIP backbone, Subject CSI1, higher is better).

Conclusion

Thank you for the detailed and thoughtful feedback. Following your suggestions, we have clarified our motivation with utilizing logit scaling in section 3.3, we have provided reasons why explained variance was used for NSD, and we have run additional experiment to evaluate different regression metrics on NSD and BOLD5000. We will update our revision to incorporate these changes.

We hope our answers address your questions, and look forward to any additional discussion.

[1] Analyzing the scale operation of attention from the perspective of entropy invariance (2021). (see line 564 of main paper).

[2] Overcoming a Theoretical Limitation of Self-Attention (2022).

[3] Qwen technical report (2023).

[4] The Llama 4 herd: The beginning of a new era of natively multimodal ai innovation (2025).

[5] Non-neural factors influencing BOLD response magnitudes within individual subjects (2022).

[6] A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision (2025).

[7] Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset (2023).