One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment

Question 1: The diversity of video stimuli data is lacking. Although the entire video database contains 1102 samples, the authors only used 102 for analysis. Why not use all of them? Since each video clip is only 3 seconds long and scene transitions are very limited, using only 102 samples makes it difficult to draw universally applicable conclusions.

Thank you for making this point. The reason for choosing only the test set of BMD is that, for RSA (our chosen method of alignment), a higher number of repetitions of the same condition (stimulus) is required to reduce the measurement noise by averaging across them for each subject. The training set has 3 repetitions and the test set 10 repetitions, with the training dataset being specifically collected for the purpose of fitting voxelwise encoding models, which is a different alignment method. Even if we were using voxel-wise encoding, the training set would only be used to train the reweighting of the models’ features to improve brain predictivity, and the brain scores on which the models would be compared would still be on the 102 test set. To showcase this point (increased measurement noise with less repetitions), we have now included a plot in the Appendix section D (Figure 17) with the RSA scores of image models on the training set (due to computational constraints, running PCA on the video models for 1000 videos would take either too much memory or too much time). It is observed that the pattern of results for the 1000 training videos (differences between image-object and image-action models) is almost identical to the results with the test set, but with a big difference in the overall scale of alignment which is almost halved. We can see that this is mainly due to the increased noise in the measurement and not because of the wider variety of stimuli, because the noise ceilings also similarly drop - so it is not only harder for the models to match the brains, but for the brains to match each other, even including the target subject - this is the Upper Noise Ceiling (UNC). A sentence explaining these results is now also added under section 3.3.

Question 2: How exactly do these comparisons motivate the design of more efficient video processing models? Based on the experimental results, it would be meaningful if the authors could give some reasonable suggestions for designing more efficient video models.

Thank you for the question - here are some initial ideas that we have on efficient design driven by these results. First, we would select the top-aligned models (with low FLOPs) as starting points. Then we can perform extensive ablations on which components specifically drive alignment - some of these model versions might be more efficient and more aligned. Second, an idea is to prune the model to the maximum point where the human alignment can be preserved. Test accuracy would also be evaluated to estimate its decrease as a result of pruning, and also compared with other pruning / efficient methods. One possible outcome is that more brain-aligned models could allow for more pruning without losing test accuracy. We have added these ideas in the discussion section of the paper.

Question 3: Is there any project code (such as a GitHub link) to ensure the reproducibility of the experiments?

We will get back to you with an anonymous link or zip in the next few days, as the workload for this rebuttal did not yet allow us to write clear instructions for reproducing the conda environment and to verify that the code runs smoothly on a new setup.

Thank you again, for your valuable suggestions. We look forward to your feedback and further discussions.

Thank you for acknowledging our systematic approach, contributions, and clear presentation, as well as pointing out important weaknesses in a constructive way. We have revised the manuscript and uploaded the new version with all changes relative to the original shown in blue. Our responses to your points one by one are as follows.

Weakness 1: The temporal resolution of the fMRI dataset. As mentioned by the authors, the low temporal resolution of fMRI is insufficient for representing the brain's processing of temporal information. It is recommended to consider using brain imaging data with higher temporal resolution, such as MEG, for this comparative study.

Thank you for bringing this up - we agree that imaging data with higher temporal resolution such as MEG from humans watching videos, or in combination with fMRI, might be more suitable for this task. This, to the best of our knowledge, does not exist yet (at least publicly). Nevertheless, even with the low temporal resolution of fMRI, this first comparative study of video models still manages to uncover differences in the models performing temporal modeling, setting better foundations for a future MEG study. We clarify this more in the updated version of our discussion section.

Weakness 2: The diversity of video stimuli data is lacking. Although the entire video database contains 1102 samples, the authors only used 102 for analysis. Why not use all of them? Since each video clip is only 3 seconds long and scene transitions are very limited, using only 102 samples makes it difficult to draw universally applicable conclusions.

Please see answer below, in the question version of this point.

Weakness 3: As mentioned by the authors, similar studies comparing brain responses to model representations on the natural scenes have been published in previous work. Although this paper tests a richer set of models, the novelty of its method and conclusions are relatively limited.

Thank you for this comment. We are aware that our methods originate from previous work on alignment in the domain of brain responses to static images. However, here we apply these to the new domain of brain responses to video, with video AI models. Obtaining the results that the established methods produce in a new domain is crucial to check their validity - in the video domain, none of the conclusions made for static images had been validated before, and it is not possible to study the influence of temporal modeling in responses to static images. After establishing a line of work for the video domain (e.g. also an MEG/EEG comparative study with the same established methods), the field can move to new alignment methods that are maybe also specific to video (e.g. a form of dynamic alignment for multiple timesteps), which would be a methodology novelty.

Weakness 4: The authors mention that this work can inspire us to design more efficient video processing models. How exactly do these comparisons motivate the design of more efficient video processing models? Based on the experimental results, it would be meaningful if the authors could give some reasonable suggestions for designing more efficient video models.

Please see answer below, in the question version of this point.

Weakness 5: Is there any project code (such as a GitHub link) to ensure the reproducibility of the experiments?

See below, in the question version of this point.

Thank you for the positive review acknowledging our contributions and quality of writing, along with suggesting literature and posing questions that concretely improve the paper. Below are the responses to your specific points; we have also uploaded an updated version of the manuscript, with changes relative to the original shown in blue.

Question 1: Figure 2a shows that image models trained on object classification are (slightly) more aligned to the early and intermediate (V3ab and hV4) visual areas than image models that are trained on action recognition, but are worse in later visual areas. I wonder if the authors have any suggestions as to why this is the case.

Thank you for this question. For the image-object models being slightly more aligned than image-action models in early/intermediate regions, this could reflect that the action recognition task is not important in early regions; the action models are first pretrained on ImageNet the same way as the image-object models, and thus the lower alignment must be due to to their finetuning on action recognition, potentially by degrading well-aligning low-/intermediate level representations. In late areas the image-action models are more aligned, suggesting that the action recognition fine-tuning benefits alignment. We have added this remark in the discussion section of the paper.

Question 2: The brackets and asterisk above the box plots in Figure 2a need some spacing between pairs of boxplots. Right now they are all on the same line and it is difficult to read which pair is which.

Thank you for this suggestion which improves the readability of the figure. We adjusted figure 2a with some spacing between the brackets of different pairs of boxplots, so that they are not all on the same line.

Question 3: The authors reported a negative correlation between models’ FLOPs (computational complexity) and alignment to higher visual areas. Does the same trend hold if we compare the model performance, maybe the validation or test accuracy (or whatever metric that was used) of the model in their respective datasets? e.g. Do more performant models in their specific task lead to worsened alignment?

Thank you for the question. We had included the relation to model test accuracy (this was for action recognition models, on Kinetics 400) in Appendix section E, as the correlations were not consistently negative or positive across brain regions. They are mostly positive in earlier processing regions and mostly negative in later regions, but in most regions the correlation was not significant. So the consistent negative relation between FLOPS and alignment is not mirrored in performance. We note this in the final sentence of section 4.

Question 4: The results show that CNNs exhibit a better hierarchy overall, while Transformers are more aligned to early visual areas. I know that ConViT is already part of the model zoo but have the authors examined architectures that use both convolution and self-attention? Such as CvT [2].

Thank you for the suggestion to investigate hybrid models (Transformers using both convolution and self-attention). We have added a new figure in Appendix D (Figure 15) to explore this. In the interest of staying within the bounds of the model libraries already utilized in the paper, we examine ConViT and a video transformer, Uniformer, that combines convolution and self-attention. We observe that the alignment progression across layers in ConViT is very similar to the other image transformers in all ROIs (subfigure a), but more interestingly, the alignment progression in Uniformer seems to lie somewhat in between that of other video transformers and video CNNs (subfigure b). This is an intriguing insight that will be good to investigate further (with more hybrid models) in future work - we mention this in our updated discussion section.

Lastly, we thank you for highlighting a paper that reflects our findings (paper 1 from your reference list), which we have now incorporated in the discussion, and again thank you for the rest of your valuable recommendations.

Question about the limitations: The paper discusses alternatives in the literature to study brain alignment via projecting feature space into voxel space. Still, it should be compared with more recent approaches than those cited in the paper and notably very successful methods that correlated brain activations and language modelling (speech/text), such as [1,2,3]. Is there any reason why those methods are performing in NLP, but maybe are not in image-based/video-based models?

Thank you for your question and the opportunity to clarify this point. We do agree that voxel-wise encoding is also a very successful method in NLP as well as in vision studies. We have updated this section to include the domain of speech as well as more recent studies.
We would furthermore like to clarify that by choosing RSA, we are not implying that voxel-wise encoding is not a successful or well-performing method. Rather, we believe that the choice between the two depends on two factors bound to the purpose of each study:

1. Performing a univariate vs. a multivariate analysis (with respect to activation arrays of brain voxels in specific ROIs, or model features)
2. Measuring the stricter emergent (out-of-the-box) model alignment to the brain vs. the more flexible (with the freedom to linearly re-weight model features) maximum achievable brain predictivity score of the model.

For studies such as reference 3 from your list, which is a reconstruction/decoding study, having a voxel-wise encoding approach is crucial as the goal is to predict the activations of the decoded sentences as accurately as possible. In this study, we were interested in a ROI-targeted, multivariate analysis and, based on our intent to perform an out-of-the-box benchmarking of AI models, we wanted the alignment measure to be stricter rather than more flexible. Having these first RSA findings (with the stricter metric) as foundations, future work can move to explore gains in predictivity as a result of feature reweighting with voxelwise modeling. We have slightly rephrased our wording in section 3.1 to further clarify this.

Methods Question 1: I understand the complexity of using high-res feature dimensionality, but it would have been nice to know/see if PCA impact the prediction accuracy (line 188) and the following results.

Thank you for this important remark. We have now included a figure in Appendix section D (Figure 16) that compares the results with no dimensionality reduction, with SRP, and with PCA. We believe the inclusion of this analysis considerably strengthens the soundness of the paper’s methodology. Briefly, this additional analysis can be summarized as follows:

1. Early Visual Cortex (EVC) alignment scores with full dimensions are lower than with PCA for all 3 model types, and all model layer depths. Especially for image models (object & action), values are much lower and more so in the middle layers, and so they fail to display the early-late hierarchy.
2. In higher-level brain regions, scores with full dimensions are more similar to PCA in the top-aligned layer (subfigure a). This is because the top-aligned layer is the last (classification) layer, which is low-dimensional anyway, so the effects of large original dimensionality do not apply. In the progression across all layers (subfigure b), scores in the early and middle layers are again lower with full dimensions than with PCA for all 3 model types.

Thus, overall, the models' RDMs match the brain less when they are constructed with full dimensions than when keeping important components. The same happens with SRP, which is a randomized choice of dimensions, still required to be large in order to be accurate. We believe that results with full dimensions / SRP can be misleading, as for example in (1) they make it seem as if the video models capture the hierarchy better, but in reality the image (object & action) model values are lower than they can be with keeping the important components. Therefore we report only the PCA reduced results in the main text.

Methods Question 2: How is the feature extraction achieved for the transformer-based architectures (line 270)? This is not clear at the moment.

Both Transformers and CNNs are treated similarly in the feature extraction. The extraction is performed with a torch extractor object that places hooks on the desired layers to be extracted. After extraction, layers’ features for all image patches are flattened, producing a single one-dimensional feature vector per layer. We have updated section 3.2 accordingly.

Thank you for your thorough review and actionable feedback. We have revised the manuscript and uploaded the new version with all changes relative to the original shown in blue. Our responses to your points one by one are as follows.

Weakness 1: The abstract and introduction lack clarity and could benefit from better writing. For instance, the concepts of "brain representational alignment" and "model-brain alignment" in the abstract are unclear and can be confusing. The second bullet point in the contributions is not very clear.

Thank you for pointing out our presentation in the abstract and introduction could be improved. In the abstract, we changed “brain representational alignment” to “representational alignment to the human brain” and “factors of variation that affect model-brain alignment” to “factors of variation in the models that affect alignment to the brain”. The second bullet point in the contributions now reads “We decouple the alignment effects of temporal modelling from those of action space optimization by adding image action recognition models as control, as well as examine the impact of model architecture and training dataset, all comparing across a fine-grained variety of brain regions.”. We went through the rest of the Abstract and Introduction, leading to a few more repairs that improve clarity.

Weakness 2: Although the question addressed is highly important/interesting, the reason for conducting such research is not sufficiently well-motivated. Particularly, it is difficult to clearly understand the neuroscience motivations for comparing brain activations to features extracted from deep learning models and how the study would benefit both the development of neuroscience and deep learning architectures in the future.

Thank you for this important feedback - we have adjusted the first paragraph of the Introduction to express the motivations more clearly as follows. “Representational alignment is a cornerstone of cognitive computational neuroscience \citep{kriegeskorte2018cognitive}, a discipline that aims to identify neural mechanisms underlying cognition by employing task-performing computational models, such as deep neural networks, for hypothesis testing. This has been shown to be a highly progressive research programme yielding novel neuroscientific insights \citep{doerig2023neuroconnectionist}. Identifying model design choices that strongly impact alignment can not only shed light on the underlying brain mechanisms, but also guide machine learning on borrowing brain's advantages such as efficiency and robustness, for example by using the top brain-aligned designs as starting points for further model development.”

Weakness 3: One of the main weaknesses is the lack of description of the data acquisition and data preprocessing, which are crucial for a paper of that scale. More extensive (even in the appendix) information about the fMRI acquisition (TR, resolutions, etc.), the preprocessing strategies, and the anatomical/functional registration strategies would be required to fully evaluate how brain signals could be reliably used for comparison. Otherwise, this would undermine the quality of the results and conclusions that can be drawn (see questions below)

We now provide a comprehensive summary of the fMRI data acquisition and preprocessing in Appendix section A (see also specific answers to your questions below).

Weakness 4: Some points of the methodology could benefit from more clarity (see questions below)

Thank you for highlighting these points - we refer you to the specific answers to your questions below.

Weakness 5: Some limitations of the study could be better discussed: choice of the fMRI dataset, and the limitations of using the Brain Representational Alignment to model brain alignment.

Thank you for this comment - our limitations section now starts with “We base our choice of alignment metric on the comparison of RSA and veRSA made in \cite{conwell2022can}, but this was on static images and it could be worthwhile to make the same comparison for our video data, as well as comparison between other metrics. RSA does not examine possible gains in predictivity as a result of linear feature reweighting. [...] Our results are based on a single fMRI dataset, and not validated across multiple fMRI experimental setups.”

Data Question 5: Are the ROIs defined subject-wise or using a template? This should greatly impact the correspondence of functional regions between subjects.

ROIs were defined using templates from three commonly used atlases, namely the Wang atlas for retinotopic ROIs, the Julian atlas (for category-selective ROIs) and the Glasser atlas (your reference 5). Importantly however, the voxels within each of these ROI masks were additionally subjected to a selection criterion based on each individual’s response in the independent functional localizer scans (50% top active voxels), thus also taking into account subject-wise variation. We now describe this in the Appendix A: ROI definitions.

Data Question 6: There is no discussion on the limit of using voxel-wise modelling compared to surface-based modelling. However, surface-based modelling has been shown to be a better predictor of brain function [5,6].

Thank you for the suggestion, we added in the limitations section the following “Surface based modelling could also be considered as a potentially better predictor of brain function \citep{glasser2016multi, coalson2018impact}.”

Finally, we would like to express our gratitude to you for providing this very constructive feedback (and taking the time to do so). We hope that with these changes your opinion of the paper can be improved, and we look forward to your answer and further discussions.

Methods Question 7: From Figure 7, it seems that the early visual cortices v1, v2, and v3 have very similar patterns, although one would expect hierarchical modelling in these particular layers. Is this the case? Is there any rational behind?

It is true that V1 and V2 look pretty similar, however V3 noticeably differs - it shows an upwards tilt in the layer lines for all 3 model types (i.e. the alignment at depth = 1 is now greater than the alignment at depth = 0). This trend continues going into the intermediate regions hV4, V3ab, with the pattern there being in between Early Visual Cortex and late areas. As to why V1 and V2 are very similar, this could be related to the measurement or alignment method not discerning the nuances of the representations in these areas.

Data Question 1: A thorough description of the dataset used is clearly missing. In particular, what is the voxel resolution, the MRI acquisition parameters, the TR of fMRI?

We have now added a summary of the dataset, including these relevant parameters, in Appendix A: MRI data acquisition. All details of the publicly available dataset can be found in Lahner et al. 2024.

Data Question 2: Is the brain solely anatomically aligned or functionally aligned? One would expect the brains to be functionally aligned, e.g. using MSM registration [4], as the study compares brain functional activation between different subjects, and anatomical registration might not be sufficient to compare brain activations at the voxel level.

We use a preprocessed version of the MRI data provided by the original authors of the BMD (Lahner et al., 2024). This preprocessing was conducted using the standard fMRIprep pipeline, which does not include advanced methods such as MSM. Following standard practice in fMRI studies, functional MRI scans containing BOLD responses to the videos were aligned to anatomical reference scans and then to standard MNI space. This last step is necessary for the application of ROIs derived from standard atlases to each individual subject’s BOLD data (see further discussion below). Importantly, however, the RSA approach does not entail any averaging or direct comparison of voxels across subjects. Instead, voxel activations for each individual subject and ROI form subject- and ROI-specific RDMs which are separately correlated with model RDMs as described in section 3.1 of our paper. We now clarify these details of the dataset in section Appendix A: MRI data preprocessing and Appendix A: ROI definitions.

Data Question 3: Are the brains aligned across repetition of the same stimuli? It is not clear if this occurs during the same session or in different sessions. As voxels are averaged across subjects, this seems important.

As noted above, voxels are not averaged across subjects, but rather RSA scores are computed separately for each subject and then averaged across subjects. However, the voxels of each subject are indeed averaged across multiple repeats of the same stimuli. This is described in section 3.1 of our paper. These repeats of the same stimuli were randomly distributed across 4 sessions (sessions 2-5), necessitating alignment of runs between sessions. This is achieved by fMRIPrep by alignment to the anatomical reference and then standard MNI space (see above).

Data Question 4: Information is missing about whether the study takes into account the temporal lag between brain activation and stimuli to compare brain activation and features. Also, the limitations of using BOLD signals to model brain processing should be mentioned.

Temporal lag is modeled in the General Linear Model that is used to estimate the brain activation to each stimulus (referred to as beta estimates) by including a hemodynamic response function (HRF). The preprocessing version from Lahner et al., (2024) that we used employs a state-of-the-art GLM method referred to as GLMsingle (Prince et al., 2022), which fits an optimal HRF for each voxel. We now include these details and relevant references in Appendix A: Brain response estimation. As for the limitations of using BOLD signals to model brain processing, we adjusted our limitations section to state “Next, BOLD signals are indirect measurements of brain activity, which in turn is an indirect measurement of the brain's representations - these are latent variables that cannot be measured.”

Thank you for the positive review acknowledging our contributions and quality of writing, as well as outlining limitations and posing interesting questions. Below are the responses to your specific points; we have also uploaded an updated version of the manuscript, with changes relative to the original shown in blue.

Weakness 1: "One significant limitation is the narrow focus on visual regions, neglecting other brain networks that are also engaged during video watching. Motion and imagery cues in videos can activate regions beyond visual areas, including those related to emotion, memory, and multisensory integration. This raises the question of why the study did not analyze the entire brain, as doing so could potentially yield more comprehensive insights and improve alignment accuracy by leveraging richer information from other relevant brain networks."

Studying other brain regions engaged during video watching, such as those related to emotion or memory, sounds indeed very promising. The reason why this study focused only on the visual system, was because we wanted to set the baseline for visual areas responding to videos first, in line with the previous work extensively comparing neural networks to these areas on static image stimuli (e.g. Conwell et al., 2022). At the same time, to study other regions we would need their ROI mapping which was not included in the BMD dataset, or otherwise move to non-ROI based analysis - this also points to exploring this in a future study, rather than this one. We have added this promising research opportunity to our future work discussion.

Weakness 2: The study only benchmarks models available in a specific library, limiting comparisons across diverse training paradigms (e.g., supervised, contrastive, self-supervised). This restricts the generalizability of findings and leaves open questions about whether models trained with different methods would exhibit distinct alignment patterns.

We agree that comparing different training objectives is an important question to address in benchmarking studies such as ours. However, here we chose to use specific libraries in order to make the study more self-contained and reproducible, and easier to systematically control one factor at a time. Outside of time limitations, one could sample models from all recent and highly cited publications uniformly to ask the intended kinds of questions. Specifically for the different learning paradigms, this could also be a study by itself that we will highlight in future work.

Question 1: Why limit the analysis to visual regions only? Motion and imagery cues in videos can activate brain networks beyond the visual cortex.

Please see our response to the “weakness” version of this point above.

Question 2: There are several recent visual/neural decoding papers (such as MindEye, MindVis, etc.). How do they compare numerically with what is presented in this paper?

While the focus of such papers is usually on achieving optimal reconstructions/decoding of the sensory inputs, and less on identifying differences in alignment for specific brain regions, these papers sometimes also report results on the numerical contributions of brain regions in the reconstruction. Depending on what is reconstructed (semantics or structure), the contributions of brain regions could also vary, similarly to the alignment of different model types (image-object, image-action, video-action). The closest decoding study to ours would be one reconstructing video from fMRI, such as [a]. In figure 7 of this paper, the authors plot the contributions of each brain region to the reconstruction of the semantics, structure, or motion of the videos, and find, similarly to our study, that intermediate and late regions are more connected to video semantics, and early regions are more connected to the low-level structure, as well as the motion of the videos. We now incorporated this comparison in our discussion section.

[a] Lu, Yizhuo, et al. "Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity." arXiv preprint arXiv:2405.03280 (2024).

Thank you again for the valuable suggestions.