Rethinking Language-Alignment in Human Visual Cortex with Syntax Manipulation and Word Models

We thank the reviewer for their positive feedback, and appreciate the engagement. We want to begin by highlighting the main change(s) made to the manuscript as a direct function of the reviewer’s feedback: In this case, the inclusion of NegCLIP in our set of multimodal model candidates (see updated Figure 2) and a substantively revised Discussion section discussing the “bag of words” phenomenon in CLIP-like models (lines 459-471). NegCLIP’s vision encoder performs at around the same level as other multimodal vision encoders in both cRSA and eRSA.

As per NegCLIP’s text encoder, we were somewhat on the fence with how to proceed, as we generally did not include multimodal text encoders in our suite of model candidates, given that these are not often used for inference or for embedding-based transfer-learning after training. (Even captioning models that use multimodal embeddings tend to use just the vision encoder, and then an independent causal language model with a token adapter for the caption generation. Similarly, and at the same time, many recent CLIP-like models now use frozen text encoders with adapters in lieu of end-to-end gradient training of text and vision simultaneously.) That being said, if results for the multimodal text encoders are of interest, we are happy to report them here, and then to include them as part of our supplementary materials if the reviewer considers them relevant.

(Please note that with respect to the response below, a last-meant versioning issue resulted in some of these updates not appearing in the most recent version of our manuscript. We apologize for this error, but fully intend to include these updates in the final version if the manuscript is accepted. We thank the reviewer in advance for their understanding!)

On the language perturbation experiment (Section 2.2) with models other than SBERT-Mini-LM6: The reviewer does make a good point that we simultaneously highlight SBERT-Mini-LM6 as somewhat of a standout, but then use it as our “representative” LLM contrast in the language perturbation experiments. In retrospect, we realize this might be somewhat confusing. This is in part because the model is representative in the sense that the average of its cRSA and eRSA scores makes it one of the closest models to the overall LLM mean across experiments -- but we realize now this is because its cRSA score is far above average and its eRSA score is slightly below average. We have updated this section (with an update lost in the versioning issue mentioned above, but which we very much intend to rectify) now to include an additional model (SBERT-MPNet, close to the average LLM in both cRSA and eRSA) to demonstrate that results hold in other LLMs as well. We provide a side-by-side comparison of the original MiniLM-L6 and the new MPNet results below:

Region | ModelVariant | MeanScore [LowerCI, UpperCI]

OTC | Bag of Nouns (SBERT MPNet) | 0.441 [0.304, 0.576]

OTC | Bag of Nouns (SBERT MiniLM-L6) | 0.415 [0.296, 0.530]

OTC | Bag of Verbs (SBERT MPNet) | 0.323 [0.207, 0.442]

OTC | Bag of Verbs (SBERT MiniLM-L6) | 0.309 [0.232, 0.386]

OTC | Shuffled Words (SBERT MPNet) | 0.485 [0.396, 0.612]

OTC | Shuffled Words (SBERT MiniLM-L6) | 0.463 [0.347, 0.580]

OTC | Original (SBERT MPNet) | 0.526 [0.422, 0.636]

OTC | Original (SBERT MiniLM-L6) | 0.546 [0.465, 0.627]

In short, the results with MPNet on the word perturbation experiments are largely commensurate with those of the MiniLM-L6 -- (and statistically, the difference of differences between the two across different perturbations is not significant). All the key findings that tell us both how little word order (and words that are not nouns) matter hold just as strongly (if not more strongly) with the MPNet variant. The key revision we intend(ed) to make to the manuscript, then, was simply to update the description of Experiment 2.2 (currently lines 266-268), to read:

• “We test these degradations in 4 models: SBERT-MiniLM-6 (the standout model in CRSA) and SBERT-MPNet (one the LLMs closest to the LLM average in both cRSA and eRSA), which both learn over sentences; count vectors, which involve no learning at all; and GLOVE models, which learn only over individual words and without hierarchical attention operations.”

I thank the authors for addressing the shortcomings of existing vision-language model embeddings in their writing. I will be maintaining my score as the shortcomings of models at representing language structure still affects what inferences can be drawn about representations in the high-level visual cortex.

At the same time, however, I want to note that the added paragraph strengthens an already robust and nuanced discussion section that places the results in context and surveys valuable directions for future work.

We thank the reviewer for their feedback and engagement with our work. Please find our point-by-point responses to your questions and concerns below:

Weaknesses:

• “more diverse datasets” / “bias in the analysis”: As much as it pains us, we do ultimately agree with the reviewer that the sole and primary dataset we use in this analysis is one of the study’s more substantial weaknesses. We take care to note this (extensively) in the discussion, but also again want clarify that our use of the Natural Scenes Dataset is both a deliberate and an arguably necessary default. Multiple publications in the past few years have made claims about language-alignment on the basis of model comparisons using only this dataset [1, 2], and one of our primary intents with this work was to show (using more or less the same paradigm as many of these other works) that what has previously been proposed as evidence of ‘language-alignment’ in the visual cortical activity measured by the NSD may not be as linguistic as it initially seems. That being said, we are fully in accord that if we want to make this claim more comprehensively and generalizably, we very much should expand the paradigm of vision / language model comparison to other datasets. As we expand on more in a point below, a recent study (published since the submission of this work) has done exactly this, with what we consider to be very incisive implications about the possibility of language-alignment in the visual brain [3].
• “Generalizability from COCO captions”: In short, we agree word for word with the reviewer on this front, and believe firmly this is one of the primary reasons why we see little difference between the vision and language models in this analysis. More on this in our discussion below of a recent paper [1] that uses more and less grounded “captions” and a targeted neurophysiology (iEEG) dataset to address this directly.
• “RSA can overgeneralize / mask differences”: While we generally agree with key parts of the reviewer’s concern here, there are a few caveats and further considerations we should take into account:
  • Spatial specificity in the brain: We agree with the reviewer that our use of a broad mask of high-level visual cortex (OTC, between ~5800 and ~8000 voxels per subject after functional and reliability-based selection) may be masking subtler differences between smaller sub-parts of high-level visual cortex (including e.g. “category-selective” ROIs such as FFA and PPA) or even voxel-specific differences. Our claim is thus very much circumscribed to a notion of “high-level visual cortex” as a computational monolith, with higher-order properties that may yield different representations at different locations on the topographic cortical map it encompasses, but which may ultimately be characterized by a finite set of broad, overarching computational principles (hierarchy, separability, invariance, consistency). Our primary conclusion, therefore is just that ‘alignment to language’ does not yet appear to be one of these broad, overarching principles (at least inasmuch as our current best-in-class, largest, most reliable and most comprehensive neuroimaging dataset (i.e. the NSD) is concerned).
    • All this being said, there may be more to the story, and as we mention in the original and revised Discussion (lines 442-451, now explicitly tagged), there does continue to be some debate about more specific ROIs and voxel-level differences that we do not arbitrate on in this work.
  • Specificity of the modeling procedure: While we do not report sub-ROI or voxel-level differences, one important consideration we do want to highlight is that that we are using two methods of RSA: unweighted, “classical” RSA (cRSA) and “voxel-encoding” RSA (eRSA). We agree with the reviewer that the first of these (cRSA) applied to a large-scale neural sector is a relatively coarse method for studying alignment. Importantly, though, our second RSA procedure (eRSA) approach is actually both a voxel-wise and layer-specific analysis procedure. eRSA involves guided re-weighting of the DNN features [4, 5], under the assumption that different voxels are likely tuned to different features, and thus, that each voxel’s response profile should be modeled as a weighted combination of the units in a layer, using independent brain data for fitting the encoding model. After fitting, each voxel-wise encoding model is used to predict responses to the test images, for which we compare the predicted population geometry to the observed population geometry of neural responses. While this last step involves comparing similarity matrices, just as in classical RSA, it is critical to note that the model-derived RDM is generated from a set of voxel-wise encoding models. Thus, the encoding procedure should be flexible enough to reveal any differences of the sort the reviewer mentions.

(Continuing from Parts 1 and 2)

Questions (Continued)

• “change with more abstract / conceptual stimuli”: One of our primary motivations in pursuing this work was a default operating hypothesis that the prediction of visual brain activity by pure-language and language-aligned vision models (observed concurrently by multiple research groups) had very little to do with language, and far more to do with the particular set of stimuli used when measuring the visual brain activity. By extension, we also assumed that this ‘alignment’ would not extend to certain kinds of brain activity that seemed a priori to involve operations well beyond the purview of either modality (vision or language) alone. (One example we would often discuss as a group were ‘gnostic neurons’ in hippocampal regions, i.e. the ‘Jennifer Aniston’ cell, with representational invariances that spanned images, text, and audio alike -- an invariance we thought could not feasibly have emerged from purely feedforward visual operations. Another example were what we would often refer to as ‘ungrounded’ concepts like ‘love’ or ‘justice’).
  • Perhaps needless to say, then, it has long been our belief that measuring the representational alignment between brain activity and vision / language models given more “abstract / conceptual” stimuli would produce a very different set of results than the ones we obtain in this particular analysis with this particular brain data (measured over COCO images and captions). And as of just two weeks ago, we may now actually have some direct empirical backing on this front thanks to the aforementioned paper [1] (a preprint published after our original submission) that contrasts the representational alignment of vision and language models on a dataset explicitly designed to elicit different levels of conceptual abstraction. In line with both our default hypothesis and our empirical results in the current work, this work demonstrates that vision and language models yield similar predictions only when they are compared on the basis of captions that refer more concretely to observable properties of the image. When they are compared on the basis of more “abstract” captions (e.g. the Wikipedia article associated with a given image), they begin to look substantially different -- with language models, especially, losing much of their alignment to the image-evoked visual brain activity. This is precisely the kind of work we hoped to inspire with our current submission, and are immensely pleased to see the field moving in this direction, and we have now mentioned / cited explicitly in our revised discussion (last paragraph, lines 511-533).

[1] Wang, A. Y., Kay, K., Naselaris, T., Tarr, M. J., & Wehbe, L. (2023). Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. Nature Machine Intelligence, 5(12), 1415-1426.

[2] Doerig, A., Kietzmann, T. C., Allen, E., Wu, Y., Naselaris, T., Kay, K., & Charest, I. (2022). Semantic scene descriptions as an objective of human vision (arXiv: 2209.11737). arXiv.

[3] “The organization of high-level visual cortex is aligned with visual rather than abstract linguistic information”, Shoham et al, BioRxiv, 2024

[4] Kaniuth, P., & Hebart, M. N. (2022). Feature-reweighted representational similarity analysis: A method for improving the fit between computational models, brains, and behavior. NeuroImage, 257, 119294.

[5] Konkle, T., & Alvarez, G. A. (2022). A self-supervised domain-general learning framework for human ventral stream representation. Nature communications, 13(1), 491.

[6] Subramaniam, V., Conwell, C., Wang, C., Kreiman, G., Katz, B., Cases, I., & Barbu, A. (2024). Revealing vision-language integration in the brain using multimodal networks. In International conference on machine learning. PMLR.

Thank you for your revision. They do address some of my concerns. I have raised my score to a 5.

There are two reasons I hesitate giving a higher score:

1. My main concern lies in the scope of the claims and if the discussion (both here in the rebuttal and the revision) are sufficient to clarify this point. I don't think my concerns around this point have been fully addressed, but I appreciate that the authors have thought about this and have made an serious attempt at addressing this.
2. I think the machine learning contribution is very limited (if any).

We thank the reviewer for the thoughtful review and constructive feedback, as well as the opportunity to clarify and improve our work. Below, we address the reviewer’s questions and concerns point by point.

(Please note, a last-minute versioning issue meant that some of the modifications we intended to include based on the reviewer’s feedback do not appear in the most recent revision. Please accept our apologies for this error. We will be sure to correct it in the final version of the manuscript should it be accepted).

Primary Concern 1: Scope / Breadth of Claims

• On limitations of current language models: The reviewer raises an important point about whether future language-only models might better capture spatial relationships or compositional semantics. We agree that this is a possibility and have clarified in the manuscript that our findings are constrained to the particular set of models we use in this analysis (lines 436-440), however representative they are of the current state-of-the-art with respect to LLM syntax comprehension. Specifically, we do not claim that language-only models cannot one day achieve such capabilities, but rather that their current performance in predicting neural activity appears limited to grounded object-level representations rather than compositional semantics.

  • We have also cited the relevant work ( $1$ , $2$ , $3$ ) that aligns with your observation about the limitations of current language models. These have been incorporated into our revised Discussion section (lines 440-471) to acknowledge this broader context.

• On fMRI modality limitations: Yet another point you raise is on the limitations of fMRI as a measurement modality and whether our claims would generalize to techniques with finer temporal resolution, such as electrophysiology or calcium imaging. While space constraints did not allow us to expand the discussion significantly beyond the main revision (updated to discuss the limitations of the language models), we initially added a new appendix (“Other Limitations of the Dataset”) with a targeted subsection to address this point directly (though please see the “versioning issue” note above).

  • "Another limitation of the fMRI dataset in our study (and of fMRI in general) is temporal resolution. fMRI is deeply constrained and slow in its ability to capture temporal signal and may not reflect transient neural dynamics that could encode more sophisticated compositional semantics. Future studies using methods such as electrophysiology or calcium imaging, which provide finer temporal resolution, could test whether such representations emerge under different temporal scales or experimental paradigms.”

• While our findings suggest limited alignment at the timescale measured by fMRI, we acknowledge the importance of investigating this question using complementary methods in future work. At the same time, we should in due diligence acknowledge that fMRI has particular advantages for studying phenomena relating to visual-language alignment in humans – the alignment between visual and linguistic representations necessitates examining distributed neural activity across broad swaths of visual and associative cortex—a level of coverage that intracranial electrodes rarely achieve. Adding to the appendix above, we address this proviso as follows:

  • "At the same time, fMRI is well-suited for this study due to its ability to provide comprehensive spatial coverage of the entire human brain, including the high-level visual and occipitotemporal cortices critical for investigating visual-language alignment. While intracranial recordings offer superior temporal resolution and direct neural measurements, their use is inherently limited by the clinical constraints of electrode placement, which often precludes comprehensive coverage of the regions most relevant to this research. Specifically, the alignment between visual and linguistic representations necessitates examining distributed neural activity across broad swaths of visual and associative cortex—a level of coverage that intracranial recordings rarely achieve. Furthermore, fMRI allows for data collection in healthy individuals rather than restricting the study to patients undergoing invasive monitoring, thereby enabling a broader and more representative investigation of these neural processes."