Transformer brain encoders explain human high-level visual responses

We thank the reviewer for their thoughtful comments and feedback! Please see below for our responses.

1) Interpreting attention maps

In this work, we focused on the NSD-general subset of areas in the brain, comprising mostly of visually responsive vertices in the back of the brain (Fig.1 A). These vertices are mostly labeled with known categorical selectivity. However, in parallel work, we have extended our model to predict the activity for the parcels across the whole brain and we have observed the effectiveness of our method in labeling unknown regions even beyond the traditional visual cortex (e.g. ROI for tool use).

However, to strengthen our claims on attention effectiveness, we have done experiments now to quantitatively examine the maps. We classified each pixel of the test set images into five categories using YOLOv5 (Ultralytics., 2020) and YOLOv8-face ( Dhar, A., 2023): background, face, body, animal and food. For each ROI, we calculated and resized the attention maps to $434 * 434$ for these images, and report the categories of the$~2k$ pixels with top attention values. We found that the category selectivity is consistent with ROI labels, with EBA most selective for body, FFA most selective for face, and OPA/PPA/RSC most selective for background.

Below shows result for subject 1, we have observed similar results with subject 2, 5, 7 and will include them in the revision.

2) Smooth gradient of feature coding both semantically and spatially revealed by attention maps

A unique aspect of our approach is that it provides learned queries that determine the type of information that would be routed to each ROI. We can therefore study the similarity of the query embeddings to study attention behavior (more similar query embeds lead to more similar attention maps between ROIs). Supplementary figures S5 - S8 (Sec A.2) show the cosine similarity between all these learned ROI queries. These similarities are also highly correlated with functional connectivity between these areas.

As the reviewer points out, the graded attention routing signal emerges along the hierarchy with areas becoming less spatial and more categorical. In early visual areas a checkerboard pattern emerges, where V1v in left hemisphere routes very similar information as the V2v in the same hemisphere (ventral ROIs more similar to one another than to their dorsal counterparts). Meanwhile, attention maps for higher visual ROIs are more semantically selective and form categorically selective clusters. We will add more discussion of this graded process in the revision.

3) For Figure 2b ,2c and 3b, showing all subjects average (instead of just subject 1) with errors bar will help readers get a sense of the improvement of performances with respect to individual variability.

Thank you for the excellent suggestion, we will include this figure in the revision.

We hope that our responses have addressed your comments and we look forward to any additional questions!

We really appreciate the thoughtful review! Below we respond to all the comments and questions.

1) Algorithmic novelty of applying the attention mechanism

As you and other reviewers pointed out, the application of the attention mechanism is an elegant solution to the problem we set out to solve that can be easily applied in many different setting and massively cuts down on the number of parameters while improving performance by taking into account the dynamics of brain computations. We believe this contribution is significant in many ways.

We also want to highlight what other algorithmic explorations will be enabled by this approach. It connects the work to dynamic routing of information, understanding selectivity, and general connectivity of brain areas, among many other important topics in computational neuroscience and AI such as how neuronal circuits are reconfigured to dynamically route information for different purposes.

2) On the effect of linear mapping between attention and fMRI activation mapping

As you pointed out, a fully connected feedforward operation is applied to all the tokens after the attention operation. In our experiments, removing this operation did not have any noticeable impact on the performance.

The main non-linearity in our model is in the softmax applied to the scaled dot products to calculate the attention weights. We observed that removing this non-linearity had a significant impact on accuracy (~0.06 points decrease).

We will discuss these points in the revision.

3) Ensemble effect

You are right that there is some ensembling effect on the performance and for that exact reason we apply this cross-validation on all the models including all the baseline models. We agree otherwise it would not have been a fair comparison. We will make this clear in the revision.

4) Additional baselines ([cls] token and PCA + Ridge.)

Excellent suggestions. We completely agree that these two baselines are important to include and thank the reviewer for pointing them out. We show their performance below in comparison to the Ridge regression and transformer attention-based encoding models.

Encoding accuracy using DINOv2 backbone

The CLS token takes a weighted average of all the important/salient image tokens to create a compact representation of the image. But mapping this token to all the vertices in the brain does not result in good performance. The big difference to note is that our model allows each ROI or vertex to dynamically route the tokens that have relevant content for that ROI or vertex so in a way each learn to create their own "CLS" token based on the content of the image and the ROI/vertex selectivity. The PCA based model performs better but still below the other two baselines. We will add these analyses to the revision.

5) Multiple datasets or data modalities in the analysis

In this paper we performed our experiments on the NSD dataset, as this is the largest and most commonly used dataset in the field, making comparison to other works easier. But we have explored a range of different encoding models, feature backbones, and routing mechanisms in this work. We also provided results in section 4.4 (Fig. 6) for the text modality. We believe these results strongly support our claims in this paper and we leave testing on other datasets to future work.

6) Values and keys in the decoder architecture

As shown in Fig. 1B, the value tensors are extracted from the feature backbone. We then create positional embeddings based on the size of the feature maps (i.e. the number of tokens). The Keys are then set as values + positional embeddings. We follow the same exact process for all the models including ResNets.

Thank you again for your comments! We hope our answers have helped clarify your questions, and we look forward to any additional discussion.

We thank the reviewer for the very thoughtful review. Please see below for our responses.

1) Neuroscientific grounding for flexible routing

Thank you for this insight. A flexible routing mechanism would indeed be reflected in deviation from the classical RF characterization of responses, so content-dependent RF shifts provide some evidence for a more flexible mechanism. Another important connection is to studies of low-rank communication between brain areas which can also be modulated by attention. We will add the following references to acknowledge the important links to the literature and would appreciate any other studies you can suggest.

Ramalingam, N., McManus, J. N., Li, W., & Gilbert, C. D. (2013). Top-down modulation of lateral interactions in visual cortex. Journal of Neuroscience, 33(5), 1773-1789.

David, S. V., Fritz, J. B., & Shamma, S. A. (2008). Task reward structure shapes receptive field modulation in primary auditory cortex. Nature Neuroscience, 11(6), 639–646.

Gilbert, C. D., & Li, W. (2013). Top-down influences on visual processing. Nature Reviews Neuroscience, 14(5), 350–363.

Womelsdorf, T., & Everling, S. (2015). Long-range attention networks: Circuit motifs underlying endogenously controlled stimulus selection. Trends in Neurosciences, 38(11), 682–700.

João D Semedo, Amin Zandvakili, Christian K Machens, Byron M Yu, and Adam Kohn. Cortical areas interact through a communication subspace. Neuron, 102(1):249–259, 2019.

2) Alternative attention baseline

Thanks for bringing the Khosla et al. (2020) study to our attention. It it very relevant and we agree the performance on this baseline is very informative.

To implement this baseline, we used DeepGaze (Kummerer et al, 2022), a state the art saliency model, to generate bottom-up saliency maps for all the images in our dataset. We then resized the maps to $31*31$ and used the resulting attention values (weights) to combine the token representations to create a single token. A linear regression was then trained to map these compressed representations to vertex activation.

Encoding accuracy using DINOv2 backbone

The table above shows the encoding accuracy for this model across the four subjects. As you can see from our response to reviewer RcVD under additional baselines, the encoding accuracy of this model is comparable to the CLS+regression model. This is not surprising as the CLS token does take a weighted average of all the other tokens that are important for classification and one would expect the important tokens for classification would also have high bottom-up activation. We will discuss the saliency-based baseline in the revision as it will indeed "serve as a useful comparison to isolate the benefit of cross-attention versus generic feature reweighting."

3) ROI-level granularity: Applying the same dynamic routing across all voxels in an ROI may obscure within-ROI variability—particularly in early visual areas with heterogeneous RFs. This might explain the limited gains of vertex-based queries in higher early visual regions.

Exactly matches our intuition as well. We will make this point clear in the revision.

4) Ways to improve the the spatial-feature factorized baseline model

Thank you for these suggestions, we did explore applying sparsity on the maps and did not see a noticeable difference in the accuracy, but we agree smoothness constraints are worth exploring as well.

5) Ensembling across layers results

The ensembling results show how the encoding accuracy would improve by taking into account features from earlier layers of the feature backbone. And since those features are coming from earlier layers, and we observe that the improvement are mostly in earlier visual areas, this shows further correspondence between brain areas and feature layers. It also highlights our method being suitable for ensembling as it is desired in many applications.

6) Quantitative examination of attention maps to substantiate interpretability claims.

Excellent suggestion. We have done experiments now quantifying this process. We classified each pixel of the test set images into five categories using YOLOv5 (Ultralytics., 2020) and YOLOv8-face ( Dhar, A., 2023): background, face, body, animal and food. For each ROI, we calculated and resized the attention maps to $434 * 434$ for these images, and report the categories of the$~2k$ pixels with top attention values. We found that the category selectivity is consistent with ROI labels, with EBA most selective for body, FFA most selective for face, and OPA/PPA/RSC most selective for background.

Below shows result for subject 1, we have observed similar results with subject 2, 5, 7 and will include them in the revision.

Thank you again for your great suggestions. Please let us know if you have any additional questions or comments!

We thank the reviewer for their very thoughtful comments and feedback! Please see below for our responses.

1) Insights from better performance

We agree that it is very important to translate better accuracy into insights about brain computation. The big insight in our work is that the encoding performance reflects whether the routing in the model is capturing the routing of information in the brain (i.e. receptive field dynamics). For the different results we highlighted this insight to give better intuitions but will do so more in the revision (e.g. an insight highlighted by reviewer EjBm: "ROI-level granularity: Applying the same dynamic routing across all voxels in an ROI may obscure within-ROI variability—particularly in early visual areas with heterogeneous RFs. This might explain the limited gains of vertex-based queries in higher early visual regions.").

2) Improved interpretability relative to other approaches.

Thank you for this suggestion. We showed attention maps in Fig. 5 and discussed how the attention behavior can be interpreted to reveal the selectivity of an ROI.

We have now run experiments quantifying this process. We classified each pixel of the test set images into five categories using YOLOv5 (Ultralytics., 2020) and YOLOv8-face ( Dhar, A., 2023): background, face, body, animal and food. For each ROI, we calculated and resized the attention maps to $434 * 434$ for these images, and report the categories of the$~2k$ pixels with top attention values. We found that the category selectivity is consistent with ROI labels, with EBA most selective for body, FFA most selective for face, and OPA/PPA/RSC most selective for background.

Below shows result for subject 1, we have observed similar results with subject 2, 5, 7 and will include them in the revision.

A unique aspect of our approach is that it provides learned queries that determine the type of information that would be routed to each ROI. We can therefore study the similarity of the query embeddings to study attention behavior (more similar query embeds lead to more similar attention maps between ROIs). Supplementary figures S5 - S8 (Sec A.2) show the cosine similarity between all of these learned ROI queries.

A graded attention routing signal emerges along the hierarchy with areas becoming less spatial and more categorical. In early visual areas a checkerboard pattern emerges, where V1v in left hemisphere routes very similar information as the V2v in the same hemisphere (ventral ROIs more similar to one another than to their dorsal counterparts). Meanwhile, attention maps for higher visual ROIs are more semantically selective and form categorically selective clusters.

We believe these two analyses strengthen our claims on improved interpretability of our approach compared to alternatives. And we will highlight them more in the revision.

3) Understanding the significance of an encoding accuracy

All the results presented in our paper are based on encoding accuracy normalized by noise ceiling, indicating the portion of the variance that is explained. So the ceiling is always at 1 (that's as good as we expect any model to perform).

4) Define vertex early

Sure, we will add the below definition and ref in section 4.1.

"In fMRI (functional magnetic resonance imaging), a vertex typically refers to a point on the surface mesh of the cortex used in surface-based analysis. It is analogous to a "voxel" (volumetric pixel) in volumetric fMRI data, but defined in the 2D cortical surface space."

Fischl, B. (2012). FreeSurfer. NeuroImage, 62(2), 774–781. https://doi.org/10.1016/j.neuroimage.2012.01.021

5) Relation to models in comp neuro models

Relational processing in the visual and other perceptual systems is thought to be implemented through lateral and large-range connectivity in the primate brain. However, how brain regions flexibly access the results of each others inferential computations is not well understood. A content-specific selective access mechanism as provided by transformer attention is probably also important for the primate brain. Our work shows that including such a mechanism in a brain-computational model improves the model's ability to predict neural responses. Although the particular mechanism for flexible routing implemented in the primate brain is yet to be discovered, our work contributes to the evidence that the human brain employs some such mechanism. We will add a discussion on this in the revision and plan to use the following references to acknowledge the important links to the literature.

Ramalingam, N., McManus, J. N., Li, W., & Gilbert, C. D. (2013). Top-down modulation of lateral interactions in visual cortex. Journal of Neuroscience, 33(5), 1773-1789.

David, S. V., Fritz, J. B., & Shamma, S. A. (2008). Task reward structure shapes receptive field modulation in primary auditory cortex. Nature Neuroscience, 11(6), 639–646.

Gilbert, C. D., & Li, W. (2013). Top-down influences on visual processing. Nature Reviews Neuroscience, 14(5), 350–363.

Womelsdorf, T., & Everling, S. (2015). Long-range attention networks: Circuit motifs underlying endogenously controlled stimulus selection. Trends in Neurosciences, 38(11), 682–700.

João D Semedo, Amin Zandvakili, Christian K Machens, Byron M Yu, and Adam Kohn. Cortical areas interact through a communication subspace. Neuron, 102(1):249–259, 2019.

6) The effect of training data size on the performance

Excellent suggestion. Our models were all trained with the training split of size 8800 images. We now experiment with training the network with different fractions of the total number of training images (550, 1100, 2200, and 4400) and display the results below.

Encoding accuracy using DINOv2 backbone

It is very encouraging to see that our approach can be trained using with as little as a few hundred samples making it suitable for smaller scale experiments as well. Also encouraging is to see that our model can achieve accuracy on par with the baseline models with a fraction of the training set.

7) Temperature parameter

In all of our experiments, we used the most common choice of the scaling value, which is $1/sqrt(d)$ and with d = 768 in our model (so ~0.036). The motivation was that this parameter in many implementations of attention layer is not easily accessible so keeping the default scaling value and optimizing other hyperparameters makes it easier to use for other researchers.

However, to address reviewer's question, we did an experiment manipulating the scaling value from 0.001 to 1.0. For values near the default value (0.01 to 0.1) the encoding accuracy was very similar to the one from the default value, but moving to more extreme values (e.g. 1.0) led to lower performance. Although not conclusive, it appears to us that the default scaling value that has worked well across many domains seems to also be near optimal in this task.

8) MISC

We will add the missing ref, make the "minus" sign more clear in the figure, and will add color bar to the attention map. Thank you for pointing those out. And we hope our responses have addressed your comments, and look forward to any additional discussion!