In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain

We thank Reviewer kyMo for the helpful review. We address specific questions below.

Q1) Positioning relative to prior work

It is great that in silico mapping of the brain has received much attention in recent years, with many great papers that we build off of. Our method offers several key innovations:

1. Massive scale. We apply in silico mapping on a massive scale with ImageNet and BrainDIVE, made uniquely possible with the transformer encoder architecture. We demonstrate that our pipeline can successfully test parcel selectivity beyond those concepts explicitly shown to the participant. To the best of our knowledge, no other study has tested image datasets on this scale nor demonstrated the capability to test concepts beyond the training set in silico.
2. Mapping of the whole brain. Previous work has focused on areas in the visual cortex (mostly V1-V4, some limited higher-level areas). We expand semantic selectivity mapping to the whole brain beyond the visual area. We discover areas with novel complex semantic selectivity beyond the classic visual area.
3. In silico verification. Our pipeline can verify selectivity hypotheses in silico by evaluating how well a label can predict ground-truth activation on a held-out set.
4. New fMRI experimental paradigm. As both image datasets and encoding models improve, our encoder-agnostic pipeline offers a way to leverage these advances to accelerate and improve the accuracy of whole-brain mapping.

Q2) Reproducing selectivity of low-level areas in the visual hierarchy

We can indeed reproduce the selectivity to low-level visual features in the early visual processing hierarchy. We optimized 32 superstimuli (16 per hemisphere) that maximally activate parcels in V1, V2, and V4 using the BrainDIVE framework. FFA is included as a representative ROI in IT cortex.

Generated superstimuli for early visual areas reproduce the classic coarse-to-fine hierarchy: V1 stimuli appear as cluttered scenes filled with dense, repetitive texture; V2 images add composite color patches and rudimentary objects; V4 stimuli reveal smoother, recognizable object forms; and FFA images almost exclusively depict close-up faces, often with clear emotional expressions. We will include these superstimuli in the revision.

To quantify hierarchical properties, we examined whether the spatial-frequency content of our stimuli mirrors classical physiological findings (e.g., [1,2]). We computed the radial average power spectrum for each image [3] and calculated the proportion of spectral power above 10%, 20%, and 30% of the maximum spatial frequency. At every threshold the high-frequency energy ratio decreases monotonically from V1 > V2 > V4 > FFA, indicating that the images that best drive higher-level areas (FFA) contain proportionally less fine-scale texture and relatively more coarse, low-frequency structure.

Note: the natural image datasets (ImageNet, NSD) did not show as clear of a V1 > V2 > V4 > FFA ordering because their spectra cluster around natural photo statistics, masking early-visual preferences.

Q3) Ensemble ablation

Encoding accuracy, CLIP backbone

Encoding accuracy, DINOv2 (ViT-B) backbone

Q4) Comparison to existing baselines

The transformer-based encoding model does outperform existing models, including those that rely on an affine transformation from model representations. As the focus of the manuscript centers around the other parts of the pipeline (which we note is encoder agnostic), we did not include those comparisons here. See [4] for a comparison of a similar encoder to past approaches, showing that the transformer-based approach greatly outperforms other pre-trained vision models with affine transformations.

Furthermore, encoders that rely on affine transformations from model representations are not feasible when scaled to make predictions for the whole brain. Building a linear encoding model for the whole brain with the same size feature maps would have 242B parameters.† The transformer-based model learns parcel queries to reduce feature size to 768, while still outperforming the full linear model. An experiment comparing the two isn't feasible to conduct but [4] compares the performance of these architectures on the visual area.

Given the recent interest and advances in encoding models, our contribution emphasizes that a sufficiently good encoder can generate "superstimuli" superior to those presented to the subject, and uncover complex semantic selectivity beyond the classic visual area.

†31*31 (patch size) * 768 (dimension size) * 327684 (voxels) = 242B parameters

Q5) Comparison with existing work

As Reviewer kyMo pointed out, decoding models (including MindEye [5]) can be used to study selectivity by manipulating neural activity in different areas and examining the observed changes in the reconstructed images. However, those decoding models have only been trained for the visual areas because they require learning very parameter-intensive mappings from the voxels to the intermediate representations needed as inputs to the diffusion models. Due to this limitation, to our knowledge, they have not yet been applied to decoding from whole-brain fMRI activity as they are not suitable for the type of experiments we are performing. Future more lightweight decoding models that can generalize to the whole brain would certainly be good baseline models for our approach. We will add a discussion of these models to the revision.

[1] David H. Hubel and Torsten N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106–154, January 1962. doi:10.1113/jphysiol.1962.sp006837. URL

[2] R. Desimone and S. J. Schein. Visual properties of neurons in area V4 of the macaque: sensitivity to stimulus form. Journal of Neurophysiology, 57(3):835–868, March 1987. doi:10.1152/jn.1987.57.3.835. URL

[3] A. van der Schaaf and J. H. van Hateren. Modelling the power spectra of natural images: Statistics and information. Vision Research, 36(17):2759–2770, September 1996. doi:10.1016/0042-6989(96)00002-8. URL

[4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high‑level visual responses, 2025.

[5] Paul S. Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan J. Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth A. Norman, and Tanishq Mathew Abraham. Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors, 2023. URL

Thank you, authors, for your response.

Your clarifications about low-level selectivity, ensemble results, and positioning relative to prior work in terms of model scaling are helpful.

I would like to encourage the authors to substantiate some of these claims, however, with further evaluation to validate the contributions. Namely:

"We apply in silico mapping on a massive scale with ImageNet and BrainDIVE, made uniquely possible with the transformer encoder architecture."

In terms of scale of the evaluation dataset, many "brain-aligned" models have been evaluated on data outside of their training distribution [1, 2], so it is still unclear to my why it is being claimed that this is only possible with the proposed architecture. Most of these types of models are inherently trained to enable in-silico experimentation on held out image data, so can be readily used with ImageNet. Please correct me if I am misunderstanding your claim here.

"The transformer-based encoding model does outperform existing models, including those that rely on an affine transformation from model representations. As the focus of the manuscript centers around the other parts of the pipeline (which we note is encoder agnostic), we did not include those comparisons here."

Given that the entire pipeline is built around this model, it still seems critical that actual evidence is provided for this. I understand that this may not be feasible with every single voxel, but a comparison to baselines on a subset of voxels (e.g., V1, V2, V4) would be valuable.

Thank you again for your responses.

[1] Conwell et al. "A large-scale examination of inductive biases shaping high-level visual representation in brains and machines." Nature Communications, 2024.

[2] Schrimpf et al. "Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?" 2020.

We are thankful to Reviewer ywAe for the detailed and helpful review. We ran additional experiments to address the concerns and incorporate these additional results in our revision.

Q1) Reproducing the selectivity of other known areas

We are able to reproduce the known selectivity for all the major known areas for all the subjects. Optimal stimuli from NSD test, ImageNet, and BrainDIVE all appear similar to those reported in the literature. We will include in our revision examples like Figure 4 (verifying aTL-faces selectivity) for each aforementioned area to demonstrate place-, body-, and word-selectivity.

Q2) Reproducing characteristics of the visual hierarchy (V1 to IT)

We optimized 32 superstimuli (16 per hemisphere) that maximally activate parcels in V1, V2, and V4 using the BrainDIVE framework. We additionally include results on FFA as a representative IT-cortex ROI.

Generated superstimuli for early visual areas reproduce the classic coarse-to-fine hierarchy: V1 stimuli appear as cluttered scenes filled with dense, repetitive texture; V2 images add composite color patches and rudimentary objects; V4 stimuli reveal smoother, recognizable object forms; and FFA images almost exclusively depict close-up faces, often with clear emotional expressions. We will include these superstimuli in the revision.

To quantify hierarchical properties, we examined whether the spatial-frequency content of our stimuli mirrors classical physiological findings (e.g., [1,2]). We computed the radial average power spectrum for each image [3] and calculated the proportion of spectral power above 10%, 20%, and 30% of the maximum spatial frequency. At every threshold the high-frequency energy ratio decreases monotonically from V1 > V2 > V4 > FFA, indicating that the images that best drive higher-level areas (FFA) contain proportionally less fine-scale texture and relatively more coarse, low-frequency structure.

Note: the natural image datasets (ImageNet, NSD) did not show as clear of a V1 > V2 > V4 > FFA ordering because their spectra cluster around natural photo statistics, masking early-visual preferences.

Q3/4) Attention mechanism criticism and suggestions for improvement

We acknowledge that the attention weights in our models tend to have a strong center bias, and suspect a few possible causes. First, the images that maximally activate a given parcel tend to have the object front and center. This is to be expected as there exist more voxels that represent centrally fixated objects. Second, the Schaefer parcellation is based on functional connectivity and does not necessarily respect categorical boundaries which can weaken the attention signal. We do agree the experiment you are suggesting would target the attention maps more directly, and we will explore them in future work.

We would like to note that the attention maps are only included in the supplementary and not mentioned in the core text. We believe it is one promising avenue for interpretability using in silico mapping methods, but not a core contribution of our study. We will include in the revision an additional interpretability metric, demonstrating with representational similarity analysis that learned transformer ROI parcel queries reproduce the resting state functional connectivity between those parcels. See our rebuttal to Q1 from Reviewer jfLo for more details.

Q5) Typos in figure and text

Thank you for pointing these out. In the Figure 4 caption, the last (b) should actually be (e). The empty reference should be to [4]. We will fix these in the revision.

[1] David H. Hubel and Torsten N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106–154, January 1962. doi:10.1113/jphysiol.1962.sp006837. URL

[2] R. Desimone and S. J. Schein. Visual properties of neurons in area V4 of the macaque: sensitivity to stimulus form. Journal of Neurophysiology, 57(3):835–868, March 1987. doi:10.1152/jn.1987.57.3.835. URL

[3] A. van der Schaaf and J. H. van Hateren. Modelling the power spectra of natural images: Statistics and information. Vision Research, 36(17):2759–2770, September 1996. doi:10.1016/0042-6989(96)00002-8. URL

[4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high‑level visual responses, 2025. URL

I thank the authors for their detailed responses and clarification. My concerns have been addressed, and I support the value of this work. I will update the score accordingly.

We are very grateful to Reviewer uJzq for the thorough and constructive feedback. We report the results from additional experiments below to address specific comments, and have included these results (and the motivation) to the revision. We look forward to further discussion.

Q1) Table 2: Concern about sampling density around activation peaks as a confound

To confirm that the superior performance of ImageNet and BrainDIVE superstimuli stems from a broader diversity of concepts present—and not sampling near peak activation—we varied the top-K image threshold used to define the concept vector as suggested.

Lowering $K$ from 32 to 1 yields results that closely mirror Table 2 in the paper, with BrainDIVE and ImageNet outperforming the NSD train concept vector by a similar margin. We observe the same pattern for $K=16,8,4,2,1$ but only report $K=4,1$ for brevity. All tables will be added to the revision.

Table 2 (revised). Fraction of parcels where the superstimulus selection process ranks NSD test images better than chance, FDR corrected, $K=4,1$.

$k=4$

$k=1$

We would also like to note that traditional large-scale experiments are not—and cannot be—optimized for activation maximization. NSD is already the most massive fMRI image dataset to date, yet the cost of collecting data on an even larger image dataset like ImageNet is (currently) prohibitive. The greater density of concept coverage is still inherently an advantage of our in silico approach.

Q2) Table 3: Same concern

We ran the same experiment reported in Table 3 for different values of the top-K threshold. Lowering the top-K threshold from 32 to 1, the results are again very similar, with ImageNet and BrainDIVE outperforming NSD train by a similar margin for all $K=16,8,4,2,1$. All tables will be included in the revision.

Table 3 (revised). Spearman's $\rho$ (mean $\pm$ std) between the model-predicted and ground-truth activation rankings on the NSD test set, averaged across parcels, $K=4,1$.

$k=4$

$k=1$

Q3) Examples of follow-up experiments

Parcels in Fig. 5 are just one way to select promising targets for follow-up fMRI studies. We suggest that experiments on these parcels would easily allow us to verify our method, to ensure that our selectivity labels are not just an artifact of the dataset or model. For these parcels, a single semantic concept (i.e., a vector in CLIP space) seems to capture the selectivity well—similar to the visual area. A simple follow-up experiment would be to present these "superstimuli" to a subject and check whether higher activation is observed in these parcels.

In addition, these parcels are promising since 1) they are visually responsive, suggesting they play a role in visual processing and it is of interest to define what these regions do, 2) they demonstrate consistent selectivity that our model can predict well (Figure 5), 3) their selectivity is not well understood nor mapped in past experiments since they lie outside the visual area, 4) many selectivities (see A.5: specific sports, kitchen appliances, social gatherings) are not known to have specialized regions, so it would be of interest to experimentally confirm.

However, this is just one method of selecting "promising" parcels, and an fMRI experimenter could apply their own criteria to select parcels using our method. For example, an experimenter interested in tool use specifically could use our in silico method to discover parcels involved exclusively in tool use, then perform follow-up experiments to test parcel activation on the superstimuli that our pipeline generates.

Q4) Clarification on the NSD-train condition

The NSD-train condition uses actual measured responses to select superstimuli, while the ImageNet and BrainDIVE images use model-predicted responses. Therefore, the NSD-train condition reflects the traditional fMRI model, where an experimenter generates selectivity hypotheses based on images that experimentally maximally activate a parcel. Since all conditions are eventually evaluated on actual responses from the NSD test set, our test is, if anything, biased in favor of the NSD-train condition. That our ImageNet/BrainDIVE-based superstimuli selection still outperforms is strong evidence of the usefulness of our pipeline.

This is an important clarification that we will include in our revision.

Q5) Clarifying main conceptual advancement

1. Massive scale in silico experimentation: "Superstimuli" from ImageNet and BrainDIVE outperform the NSD training set on predicting parcel activation on a held-out image set, demonstrating that our pipeline can successfully test parcel selectivity beyond those explicitly shown to the participant. To the best of our knowledge, no other study has tested image datasets on this scale nor demonstrated the capability to test concepts beyond the training set in silico.
2. Mapping of the whole brain: Previous work has focused on areas in the visual cortex (mostly V1-V4, some limited higher-level areas). We expand semantic selectivity mapping to the whole brain beyond the visual area. We discover areas with novel complex semantic selectivity beyond the classic visual area.
3. In silico verification framework: Our pipeline can verify selectivity hypotheses in silico by evaluating how well a label can predict ground-truth activation on a held-out set.
4. New fMRI experimental paradigm: As both image datasets and encoding models improve, our pipeline offers a way to leverage these advances to accelerate and improve the accuracy of whole-brain mapping.

Q6) Interpreting rank correlation magnitude

Parcels in the visual area with known high-level selectivity (EBA, FFA, FBA) tend to exhibit very high correlations, while parcels with known lower-level selectivity (V1–V4) tend to exhibit low correlations. The average Spearman's $\rho$ from our concept vectors are in the range of areas like PPA and RSC—both widely studied and accepted in the literature.

Using CLIP as our metric allows us to capture high-level features that can explain semantic selectivity. Given that fMRI studies tend to rely on experimenter-curated topics, the CLIP space is a move toward a data-driven approach, but ultimately any metric can be substituted.

We provide baselines from known areas to contextualize our results. Since selectivity is a graded response and not binary, we don’t believe it is appropriate to have a minimum threshold that constitutes meaningful selectivity. Some parcels are naturally more visually responsive than others, and our paper lays out a method of systematically making and ranking semantic selectivity predictions. Notably, the CLIP metric allows us to rank parcels by how well we can make such predictions, which paves the way for in vivo fMRI studies of promising parcels.

Spearman's $\rho$, actual vs. predicted activation ordering in high level areas, subject 1:

Spearman's $\rho$, actual/predicted activation ordering in low level areas, subject 1:

Note that since we’re using the Schaefer-1000 parcellation, for each visual area, we average the rank correlation value across the top 3 Schaefer parcels with greatest overlap with that visual area.

Q7) Cosine similarity analysis of concept vectors for cross-subject parcels

Great suggestion. We do find that the concept vectors generated for the same parcel, when analyzed across subjects, tend to be more similar (0.784 ± 0.069 for BrainDIVE) than the parcels coming from around the target parcel (0.710 ± 0.115 for BrainDIVE). Absolute cosine similarities are high because these parcels are selective for high-level visual concepts that share similar features, but our method can adjudicate between them using targeted stimulus generation. We will add this analysis to give further intuition on the concept space of the parcels.

Q8) Clarifying construction of concept vector in cross-subject analysis

Yes, the concept vector for cross-subject experiments is generated by taking the simple average of the CLIP vectors for the top-32 maximally activating images from each subject. For any subject, this concept vector is an average of 96 CLIP vectors (32 per subject * 3 subjects).

Q9) Ambiguity of the term "pipeline"

The application of this encoder to the whole brain is a contribution of the paper (section 4.2), but you’re right that the core contribution is the steps around the encoder. Thank you for pointing out the ambiguity; we will clarify this in our revision.

Q10) Typos in figure and text

Thank you for pointing these out. We will fix these in our revision.

We sincerely thank reviewer jfLo for recognizing the strengths of our work and for the thoughtful review. We especially appreciate the positive evaluation of our manuscript. We include answers to specific questions below.

Q1) Mapping the parcels within a distributed network

Taking advantage of our transformer-based approach, we map the connections between parcels by examining the representational similarity of learned ROI queries, each of which corresponds to a single parcel. High cosine similarity between ROI queries (queries averaged across ensemble models) suggests that two parcels are highly connected, as both attend to similar content in an image. Indeed, this similarity matrix for the visual areas closely replicates the functional connectivity matrix from the Schaefer parcellation (derived from resting-state correlated responses) [1]; the Pearson correlation between the two matrices is $\approx 0.5$. We will include these results and the corresponding figures in the revision.

Furthermore, generated superstimuli for early visual areas reproduce the classic coarse-to-fine hierarchy: V1 stimuli appear as cluttered scenes filled with dense, repetitive texture; V2 images add composite color patches and rudimentary objects; V4 stimuli reveal smoother, recognizable object forms; and FFA images almost exclusively depict close-up faces, often with clear emotional expressions. We will include these superstimuli in the revision.

To quantify hierarchical properties, we examined whether the spatial-frequency content of our stimuli mirrors classical physiological findings (e.g., [2,3]). We computed the radial average power spectrum for each image [4] and calculated the proportion of spectral power above 10%, 20%, and 30% of the maximum spatial frequency. At every threshold the high-frequency energy ratio decreases monotonically from V1 > V2 > V4 > FFA, indicating that the images that best drive higher-level areas (FFA) contain proportionally less fine-scale texture and relatively more coarse, low-frequency structure.

Note: Natural-image datasets (ImageNet, NSD) do not show a clear V1 > V2 > V4 > FFA ordering because their spectra cluster around natural photo statistics, masking early-visual preferences.

Q2) Alternative parcellations beyond Schaefer-1000

We reported results from the Schaefer-1000 parcellation since it is commonly used in the literature, but we also ran analyses on a functional parcellation generated by grouping voxels with correlated responses (applying k-means on the training set, which yields a subject-specific parcellation). For each subject, the whole brain was separated into roughly 1000 parcels, each containing 200–400 voxels based on correlated responses. We observed very similar results when applying the same analyses. Parcels that overlap significantly with known visual areas demonstrate the selectivity of that area, reproducing the sanity check in 4.3, including face-, place-, word-, and body-selective areas. Many visually-responsive parcels outside the visual area also demonstrated consistent semantic selectivity of complex concepts similar to those observed in the Schaefer-1000 parcellation (social interactions, specific sports, family, etc.). Perhaps surprisingly, there were some concepts that parcels were selective for that appeared in one parcellation but not the other—though we only observed this qualitatively. These results ultimately were not included in the final manuscript for brevity and because Schaefer-1000 is a more widely-used parcellation approach.

Q3) Quantitative increase in activation by superstimuli

On average, our model predicts that the BrainDIVE-optimized superstimuli activate the parcel more strongly than top NSD test images, but not more than the best-performing ImageNet images. But for roughly a third of parcels, BrainDIVE-optimized superstimuli do outperform ImageNet, suggesting that some parcels may be selective for a concept that is not easily captured by the space of natural images (ImageNet).

Here, we report the average activation magnitude predicted by the model for NSD-test images, top ImageNet images, and top BrainDIVE images.

Model-predicted activation magnitude for top-K superstimuli from all datasets, averaged across parcels, subject 1.

Q4) Low prediction accuracy in non-visual areas

Since many of the parcels considered here are deeper in the processing hierarchy, they are selective for more abstract concepts and are driven not only by a single sensory input but also multi-modal inputs and other cognitive processes (such as attention and memory). As a result, the brain responses do not always have high agreement for different presentations of the stimuli, leading to lower signal-to-noise (SNR) ratios. For this reason, we only selected parcels with an SNR above a certain threshold for further experimentation, to make sure their visual responses are strong and consistent and ensure generalizability of the observed categorical selectivities to future experiments.

Q5) Typo in figure 3

Thank you, we have added (a) and (b) labels for Figure 3 in the revision of our work.

[1] Ru Kong, Yan Rui Tan, Naren Wulan, Leon Qi Rong Ooi, Seyedeh-Rezvan Farahibozorg, Samuel Harrison, Janine D. Bijsterbosch, Boris C. Bernhardt, Simon Eickhoff, and B. T. Thomas Yeo. Comparison between gradients and parcellations for functional connectivity prediction of behavior. NeuroImage, 273:120044, June 2023. doi:10.1016/j.neuroimage.2023.120044. URL

[2] David H. Hubel and Torsten N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106–154, January 1962. doi:10.1113/jphysiol.1962.sp006837. URL

[3] R. Desimone and S. J. Schein. Visual properties of neurons in area V4 of the macaque: sensitivity to stimulus form. Journal of Neurophysiology, 57(3):835–868, March 1987. doi:10.1152/jn.1987.57.3.835. URL

[4] A. van der Schaaf and J. H. van Hateren. Modelling the power spectra of natural images: Statistics and information. Vision Research, 36(17):2759–2770, September 1996. doi:10.1016/0042-6989(96)00002-8. URL

We thank Reviewer yocx for the constructive review and for bringing our attention to missing citations for relevant works. We regret not citing them in the original submission. These papers have been seminal in laying the groundwork for in silico brain mapping and the foundation of our methodology, and we will include citations to these papers in our revision. However, we believe that our work significantly expands upon past work and clarify the novelty and contribution of our approach below.

Clarification of a detail in the review

Reviewer yocx wrote "images selected by their approach tends to yield semantically meaningful cluster (as characterized by CLIP embedding) such that images closer to this embedding in the CLIP space tends to elicit strong response in the encoding model."

We just want to clarify that our images closer to this embedding in the CLIP space tends to elicit strong response in the ground-truth held-out set, not only by the encoding model—which allows us to validate the robustness of our predictions in silico without encoder bias.

Q1) Technical contributions

We share your optimism about the promise of AI models for in silico experimentation to study neural populations. However, past approaches face several key limitations that our work addresses, as pointed out by other reviewers.

• Reviewer jfLo highlights the "massive scale" upon which our pipeline generates and tests hypotheses as a "significant methodological advance for the field."
• Reviewer uJqz cites cross-subject generalization in evaluation as a key strength of the method.
• Reviewer kyMo raises the capability to identify selectivity outside canonical visual areas as a key "proof-of-concept" result.
• Reviewer ywAe writes that the use of encoder-decoder transformers to synthesize optimal stimuli from massive datasets "suggests a new approach for discovering visual selectivity, offering a more flexible and scalable alternative to traditional fMRI experiments."

Below, we address the novelty of our manuscript in comparison to the specific papers mentioned.

Q2) Significance compared to past work in NHP/mice

We thank the reviewer for pointing out our blindspot regarding similar works done in mice/NHP and regret not citing these studies. We believe those works laid the foundation for much of the more recent work in human imaging and we have updated the revision to include the relevant citations and past methods. Those works [1, 2, 3] have clearly demonstrated the usefulness of stimulus optimization and how they can be paired with encoders to study neural populations. The generated "superstimuli" also have been experimentally validated, which is a motivation for in silico experimentation that we have explored in this work.

We do however want to emphasize that extending those works to human neuroimaging comes with many new challenges that remain unaddressed. Namely, building a scalable state-of-the-art encoding model for the whole brain, using the encoder for exploring massive image sets and generative models for maximally activating stimuli, and creating robust methods to quantify selectivities and to validate them empirically in silico—have all not yet been explored. Notably, building a linear encoding model for the whole brain with the same size feature maps would have 242B parameters.† Our transformer-based model learns parcel queries that reduce feature dimension to 768 while still outperforming the full linear model. An experiment comparing the two isn't feasible to conduct but prior work comparing performance on the visual area supports this claim [4].

These innovations move the field from single‑site discovery ([1] and [3] looks at primate and macaque V4; [2] looks at monkey V1) to systematic whole‑brain mapping and to statistical validation without additional scanning time, which, to our knowledge, has not been demonstrated previously.

As a result of this undertaking, we have discovered visual selectivities to high level concepts that are unique to humans, such as sports and complex tool use, that could not have been studied in non-human animal models. We believe addressing all these challenges, and the discovery of new selectivities worthy of future experiments, are all novel aspects of our work in comparison to the papers highlighted by the reviewer.

†31*31 (patch size) * 768 (dimension size) * 327684 (voxels) = 242B parameters

Q3) Similarity metric defined on encoding model

We agree that we would be able to use encoder predictions to create a similarity measure, and that would be a good test of the encoder performance. However, the main goal of our methodology is to create labels for the selectivity of unlabeled parcels. Since the CLIP space provides an aligned visual-text embedding space, creating our "concept vector" hypothesis in that space can give us a semantic description of the selectivity. This also introduces a bottleneck that can lead to findings that can better generalize to future in vivo experiments.

Q4) Contrastive approach for generating superstimuli

This is an interesting suggestion for further teasing apart the selectivity of a parcel. We performed some further analyses on parcels in the labeled area, and found that even without any selection, stimuli which activate one area generally do not activate other (non-related) areas. For example, the top 32 ImageNet images that maximally activate aTL-faces already perform poorly in activating other parcels such as PPA (top 82.3%), OPA (top 81.5%), and RSC (top 62.3%).

In some cases, this approach may actually make it more difficult to correctly label the selectivity of an area. The images that maximally activate aTL-faces unsurprisingly also activate FFA (top 23.6%) and OFA (top 9.4%). Naïvely applying the suggested approach, we can generate images that activate aTL-faces but not FFA, which end up being non-face images—not the expected selectivity for aTL-faces. Of course, for labeled parcels, we could manually avoid suppressing the activation in other related areas when choosing optimal stimuli, but this fails for areas outside the visual cortex since they are unlabeled.

Generally, we believe that this approach may be useful for experimenters interested in very fine-grained analyses of specific parcels, but it is challenging to apply to whole-brain mapping. We will include these results and details about this suggested approach in the revision.

Q5) NSD acronym typo

Thank you, we revised our paper to use the full name upon first mention.

[1] Pouya Bashivan, Kohitij Kar, and James J. DiCarlo. Neural population control via deep image synthesis. Science, 364(6439):eaav9436, May 2019.

[2] Edgar Y. Walker, R. James Cotton, Wei Ji Ma, and Andreas S. Tolias. A neural basis of probabilistic computation in visual cortex. Nature Neuroscience, 23(1):122–129, January 2020. ISSN 1097‑6256, 1546‑1726.

[3] Paweł A. Pierzchlewicz, Konstantin F. Willeke, Arne F. Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, Andreas S. Tolias, and Fabian H. Sinz. Energy Guided Diffusion for Generating Neurally Exciting Images. Advances in Neural Information Processing Systems, 36, 2023.

[4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high‑level visual responses, 2025.