Revealing Vision-Language Integration in the Brain with Multimodal Networks

Thank you for your review.

I don't think this is an appropriate venue for the paper. There's no clear methodological advance in ML that would be of broad interest to the ICLR community: it needs to be read by neuroscientists, not ML people. I looked at the neuroscience papers at ICLR in the last 2 years and found only one that would classify as investigating task-driven neural networks in brains in the style of Yamins, DiCarlo, Kriegeskorte, etc., and that paper showed a clear methodological contribution (https://openreview.net/forum?id=Tp7kI90Htd). The authors should look at where this type of neuroAI work is typically published, e.g. NeurIPS, SVRHM, cosyne, PNAS, Nature Comms, etc.

First, we would like to mention that task-driven neural networks in brains have been investigated, even last year in ICLR 2023 (https://openreview.net/forum?id=KzkLAE49H9b). Furthermore, papers that have non-ML takeaways have been accepted to ICLR, both with the paper we linked and other papers in the neuroscience and cognitive science category(https://openreview.net/forum?id=xmcYx_reUn6 ).

We believe this paper could be of great interest to the ICLR and neuroscience community and has a strong methodological contribution for any person interested in understanding how well their neural network computation matches computation in the brain. For the neuroscience community, the obvious takeaway is the identified brain region. For the greater deep learning community, there is a greater interest in connecting DNNs with the brain. Specifically,

1. There are numerous attempts, like BrainScore, in neuroscience and ML to make models more brain-like so as to have the robustness of humans. By showing a link between multimodal models and the brain, we can extend this search to multimodal architectures.

2. Traditionally, the ML community has had one objective: maximize benchmark performance. Recent language models and multimodal models excel at existing benchmarks making progress increasingly difficult. We enable comparisons against the brain, providing a useful new objective function and a specific statistical methodology to analyze the objective function on our data.

3. Combining 1 and 2, at the next ICLR, you could see a modified multimodal Transformer that is far more similar to the brain. This would not be possible without our work, both in establishing what exists now and laying down a coherent methodology for doing this research.

4. The novelty of our methodology is in the “Confidence Intervals across Time” and “Model Comparisons” sections. Had we carried out this work with the standard analysis, like in Goldstein (2021), ignoring the fine-grained statistical controls we added, we would easily have found numerous multimodal areas, but they would not have been statistically meaningful. There was little need for prior papers to address this methodology because fMRI does not have high temporal resolution and Goldstein (2021) didn’t have the same kind of model comparisons.

5. Practically, this also shows that multimodal Transformers are likely on to something important. A line of evidence for their architecture that is totally divorced from any engineering benchmark (we as a community design benchmarks so that our models have some hope of handling them, so this line of evidence is somewhat contaminated) and relies only on comparing them to the brain (which we cannot influence as a community).

There are many potential takeaways from our analysis beyond the presentation of regions in the brain that we can list that may be of stronger interest to a deep learning audience in the ICLR community. For example, we could use our comparisons to identify the current best multimodal model across all multimodal electrodes. We include this as a new subsection in Section 4.3 and a new figure.

Thank you for your thoughtful review. We respond to the points below.

Model performance seems to be low. Figure 2a suggests that the Pearson correlation between model predictions and the true neuron activity is about 0.1. This means the model explains about 1% of the variance of the data (linear model r2). This is pretty low. Please explain why we should care about a model with limited prediction power.

Note that Figure 2a averages the Pearson correlation over many models, neurons (electrodes) and time bins so this is not entirely representative of the meaningful range of possible Pearson r values. Recall that our window size extends from -2s to 2s and predicting activity in a time bin close to the tail end of this window size gets very low Pearson correlation. We are including these in the average in order to have a fuller representation in the plot. As stated in Section 4, we can obtain max Pearson correlation values of ~0.52 for particular time bins and neurons.

The performance difference between brain regions, or between models is low, compared to the error bar per condition (figure 2a, and in multimodality tests). By the way, it is not clear to me what the error bar stands for. Is the difference between models or between electrodes sufficient for discrimination?

Apologies for the confusion. We should have a better caption for Figure 2, which now has been moved to Figure 7 in the appendix. Once again, the error bar in Figure 2a is generated from averaging the raw Pearson correlation scores from running a single base regression. All scores in section 4c refer to the intervals that we obtain from bootstrapping analyses so this range will be different. Based on the differences we report in this section, we believe that the difference between multimodal and unimodal models is significant. Multimodal models outperform unimodal models by 0.07 points on the language-aligned dataset and 0.09 on the vision-aligned dataset.

Whether the model selection is consistent with the known biological function of the brain areas.

For selecting unimodal models, we selected models with general architectures that were used in prior work extensively such as vision transformers (SLIP-SimCLR, BEIT), CNNs (ConvNeXt), or language transformers (SBERT, SimCSE). These models have been shown to fit the brain consistently. For our multimodal selection, we stayed with the general motif of transformer architectures which generally have seen success at modeling activity in the brain. We selected the best-performing SOTA DNN models of vision-language processing at the time. Connecting DNN computations and circuitry to neural computations and circuitry is still an open question but newer work has shown connections between the transformer architecture and the brain.

Do all multimodal tests identify the same set of multimodal selection electrodes? Quantitative results should be provided. A standard procedure should be followed to identify brain areas as potential vision-language integration regions.

Our dataset-specific tests of multimodality (test 1 and 3) do identify potentially different electrodes which is why we take an intersection between electrodes that pass test 1 for the language-aligned dataset and vision-aligned dataset for test 2 (and do the same between test 3 and test 4). We are a bit unclear on how this isn’t a standard procedure or what quantitative results the reviewer wants to see. We believe we are using a standard procedure (bootstrapping model performance difference) as a way of comparing models over the significant time bins -- this is a general procedure.

How the regression model is trained? What is the input and what is the output? For each model with a different feature size, how does the parameter space for the ridge regression model differ between models? Whether the model performance was affected by the number of parameters?

Sorry for the confusion! We do discuss these details in Appendix C -- due to space this discussion can’t find entirely in the main paper. To summarize, the regression model is trained as follows. We have a set of image-text pairs, $(i, t)$. We feed the pair (or one input in the tuple) to our DNN model, $M$, to get $r = M(i, t)$, where r is the activations of the DNN in response to $(i, t)$. We also extract a 4s window of sEEG activity from one electrode e and split this into 161 time bins of averaged activity, $e_t$. The regression is trained to take $r$ as input and predict $e_t$ for each time bin across all electrodes. So, the number of parameters in the regression is dependent on the dimensionality of $r$ which is affected by the dimensionality reduction procedure. We found that the number of parameters in the regression did not affect the regression performance across models.

Thank you for your thoughtful and detailed review! We address your points below.

Weaknesses:

(1) It would be helpful to include a better explanation of what the "event structures" are exactly. Maybe a picture.

Apologies, we realize the concept is a bit confusing. We include a figure in the appendix showing event structures selection in Appendix A.2.

(2) Page 6, Section 4, second paragraph: This bit of text here is a bit too overloaded with number and while the authors might expect their readers to be careful and try to understand what is the meaning and significance of those numbers being what they are... but a reader is rarely that careful. I would advise to add explanations to this paragraph that explain to the reader what they are supposed to think when they see this or other set of numbers. Tell the reader what they are supposed to with those numbers.

Thank you! We have updated the results section to better address the numbers here. Our goal was to present the results without overselling/overclaiming our findings but we realize that our current writing may come off as dry or needlessly confusing to a reader.

(3) Figure 2: The caption does not explain the figure well. Panels (a) and (b) are not mentioned in the caption. An attempt to explain with "mid left" and "bottom right" is confusing, perhaps just put the names of the models on the figure plots. The dots on the freesurfer brain surfaces on the right of the figure are not explained at all - what are they? For the colour-blind it is very hard to see the red dot, I recommend using blue.

Thank you! We have decided to focus on some new results and have moved Figure 2 to the appendix as Figure 7. In addition, we have updated the caption and figure.

it would help a lot to see those electrodes on a picture of a brain!

This makes sense! We now include Figure 4 which includes a visualization of the most predictive multimodal model on all of the multimodal electrodes. We hope this better addresses this comment.

(7) In your own estimation, are those singular electrodes shown on Figure 8 as "multimodal across vision and language" provide sufficient evidence to multimodal processing in those areas?

The improvement in Pearson’s r that we achieve is significant. We introduce many controls on the dataset, architecture, and statistics. We still achieve significant gains from multimodal models, sometimes close to r=0.1. We agree with the reviewer that our current draft may leave the reader feeling confused as to whether our results are trustworthy so we include some new text focusing on this. In general, we approach these results with caution due to the fact that little is known about multimodal integration, especially with vision and language. We are excited to see our results match prior work. We are also excited to see that our results are not explained by better task performance. We believe that this is a strong start to deeper exploration into these areas and what in our networks drives our prediction. That being said, our motivation for showing numbers was not to deter readers or reduce confidence but instead to be factual. We believe these studies are invaluable given the lack of understanding around multimodal integration.

Questions:

(1) The implantation sites of sEEG electrodes in your dataset were clinically motivated. To what extent did they cover the areas you were interested in? Both whether all of the areas of interest were covered, and also among the the areas that were covered - was the coverage sufficient for you analysis in your estimation and why?

Great question, and this likely connects with the weakness you were bringing up. There were some areas that were not covered in the implantation such as early visual cortex. These are areas where we know significant visual processing occurs but these could not be included in our analyses. We also had fewer electrodes in the occipitotemporal cortex or the superior frontal cortex than we would have liked, making the percentage of multimodal electrodes small in these regions as shown in Figure 4.

Figure 5 shows that we have broad coverage across the brain but a majority of the electrodes naturally fell into the superior temporal and middle temporal cortex -- one subject had all of their electrodes in this location. We had coverage across all desirable locations of the brain other than EVC and having more coverage in OTC or the superior frontal cortex would have given more evidence for these regions to be multimodal.

Thank you for your thoughtful review. We address your suggestions below.

One test that can also be included is to randomly input one of the modalities in a multimodal model and then compare with predictions using actual multimodal inputs. This test can reveal the importance of multimodal information avoiding the confounds due to architecture, parameters, training set, etc.

Thank you for this suggestion! Due to the time constraints and limited resources, we could only perform this permutation for 1 multimodal electrode and one model. This test only applies to architecturally multimodal models such as BLIP, which we use in this test.

More specifically, we randomly permuted one modality in our event structures 1000 times and fed the new inputs to BLIP and predicted activity in the 1 multimodal electrode. This led to four types of permutations: language-aligned dataset where we permute the language modality, language-aligned dataset where we permute the vision modality, vision-aligned dataset where we permute the language modality, and vision-aligned dataset where we permute the language modality. We calculate the p-value for each comparison in this permutation test. We find that for the language-aligned dataset, when we permute either modality, we obtain a p-value of 0.001 (meaning none of the 1000 scores is larger than the unpermuted score). For the vision-aligned dataset, our permutation passes (p-value < 0.05) when we permute the vision modality but not when we permute the language modality (p-value = 0.169). We have two reasons for this. For the language-aligned dataset, BLIP is the best model that explains activity in the electrode. In the vision-aligned dataset, the best model is SLIP-Combo. In general, architecturally multimodal models are worse at modeling vision-language integration than the trained multimodal models like SLIP-CLIP or SLIP-Combo. Furthermore, the alignment presented in the vision-aligned dataset is weaker. Language is not consistently paired with the main visual stimuli, scene-cuts. Our architecturally multimodal models may not be good enough to model this form of integration. We hope to run a larger version of this test but the computation is expensive and requires us to reload embeddings 1000 times for every subject and run 1000 regressions across all electrodes. This test may be better with newer architecturally multimodal models such as BLIP-2.

Did the authors perform SLIP-CLIP vs SLIP-SimCLR vs SLIP-Combo comparison? Because SLIP-CLIP is also multimodal I am curious what was the motivation for only showing SLIP-CLIP vs Combo results in Figure 4.

This is a good point! We could have included SLIP-CLIP as well but chose SLIP-Combo and SLIP-SimCLR as our main comparison. The choice between SLIP-CLIP and SLIP-Combo as our multimodal choice was arbitrary. We show the results between SLIP-CLIP and SLIP-SimCLR for comparison in Appendix H.

It is not clear to me why language alignment and vision alignment event structures leads to difference in results. I would like to read author’s explanation on why results depend on how event structures and if this is a limitation of this approach.

This is an important point and we add a few sentences to Appendix A.2. Event structures are a first-order approximation for choosing image-text pairs without imposing a strong hypothesis on the structure of real vision-language integration. Vision-aligned data selects scene cuts and the closest sentence and this may be modeling longer-term integration of vision and language than the language-aligned dataset which uses context segments and frames. This is because the vision-aligned data is less aligned than the language-aligned data since a frame occurs with every word but a sentence can occur much later (sometimes, 2 seconds) after a scene cut. We believe the event structure selection has strengths and limitations. Our event structures are naturalistic and intuitive, prioritizing one modality and sampling the other modality. They do not rely on model architecture or make assumptions about vision-language integration. They also cover many forms of possible integration by strictly aligning (language-aligned) and loosely aligning (vision-aligned) the modalities. However, we recognize that in an ideal world, we would have defined vision-language events similar to words or objects that are easily extractable from our movie dataset where we have reasonable evidence that vision-language integration is occurring. In the end though, we argue that the event structure selection makes for a more difficult benchmark by greatly expanding the potential forms of vision-language integration that multimodal architectures need to consider.

Weakness 4:

The contribution to an ML audience is not very clear. I believe this work will be better suited to be evaluated by neuroscientists, since the claimed contributions are on the neuroscience side, and will also be more appreciated at a neuroscience venue.

We believe that there is a core contribution to the ML audience for comparing neural networks with the brain. We design a standard pipeline with statistical comparisons that lend credibility to a conclusion that one model fits the brain better than another. We highlight our ML contributions in an updated subsection 4.4. In Figure 4 we show the best multimodal model across our multimodal electrodes that we find. We believe this investigation can lead to natural follow-up work with our data and pipeline such that in future years, new papers with new multimodal transformers can be released and these architectures are hopefully more similar to the brain. We also want to point out that prior papers have been accepted to ICLR despite not having considerable ML contributions such as [1] or [2]. We believe our work is in-line with what is presented at ICLR.

[1] Wang et. al. BrainBERT: Self-supervised representation learning for intracranial recordings. ICLR 2023.

[2] Loong Aw and Toneva. Training language models to summarize narratives improves brain alignment. ICLR 2023.

Thank you for the review. We address your points below.

Weakness 1: We address this point-by-point below.

The biggest weakness is the central claim that the proposed analyses of multi-modal models can localize vision-language integration in the brain. Can the authors please define what they mean by vision-language integration?

This is a good point. We define vision-language integration to be any non-linear combination of vision-language features as carried out by the candidate neural networks. Our candidate DNN models like ALBEF, Flava, BLIP, SLIP-Combo etc. represent a hypothetical integration of vision-language features. Effectively, we are testing whether these DNNs model activity in the brain better than language or vision models.

Using neural networks in this way is desirable. We are not attaching ourselves to a specific hypothesis of vision-language integration but instead allowing us to have a larger search space.

Even on the model side, it is an open question to what degree multi-modal models actually integrate information from multiple modalities as opposed to increasing the alignment between individual modalities (Liang et al. 2022 https://arxiv.org/pdf/2209.03430.pdf). It is not clear whether the results that are observed are due to vision-language integration or whether they are due to improved representations of the language-only or vision-only modality.

Could the reviewer explain what they mean by “increasing the alignment between individual modalities”? How is this distinct from integration? The citation the reviewer provides also does not seem to provide any evidence for their claim as we understand it -- Section 4 gives evidence that models perform some alignment between modalities that allows for improvement on many tasks that require some form of multimodal reasoning such as visual question answering, etc. The paper doesn’t show that better unimodal representations are necessarily what leads to improved performance on tasks VQA -- if this was true, architectures like LXMERT or VisualBERT would still be SOTA as long as we introduce better visual or language representations which is not the insights that led to models like BLIP, ALBEF or Flava. Furthermore, we are unsure how our results don’t dispel this doubt. We show that multimodal models significantly out-perform both vision and language models at predicting vision- and language-aligned data. We show that these models do not have strictly better language or visual reasoning, even on our dataset in Table 5 and Table 6. We include a combined version of Table 5 and 6 which presents average language and vision task performance and a new subsection discussing this in Section 4.3. We also include Tables 5 and 6 below.

Thanks for the response. My main concern still stands. The authors now clarify that what they mean by vision-language integration is "any non-linear combination of vision-language features". However, any non-linear combination of vision-language features can also include a mapping in which the vision features are even exactly the same as they were before the non-linear combination but the language features are combined in new non-linear ways. Would the authors call this integration? I would not. This is related to the thought experiment I proposed. If I apply the authors' methods to study different layers of a language-only encoder, which recombines the previous layers in non-linear ways, I will find that some layers predict brain responses better than other layers. But this is clearly not due to any multimodality.

I am keeping my score. I recommend that in an improved manuscript the authors clarify their integration definition and crisply explain how their results support the drawn conclusions. I also recommend clarifying the contributions to an ML audience. The papers that were referred to either have clear methodological contributions (Wang et al.) or interpret ML models (Aw et al.).