Visual Representations in Humans and Machines: A Comparative Analysis of Artificial and Biological Neural Responses to Naturalistic Dynamic Visual Stimuli

Weaknesses: Overall, the paper is confusing as written, and this results in difficulty understanding that muddies what is I believe otherwise exciting work. For example, many of the results are simply reported with little to no summarization or interpretation of the larger significance & meaning. There is an overall lack of justification for the methods chosen, I believe due to lack of relevant citations to previous work. For example, I could not find a justification for the author’s choice of comparison metric for model/RDM. The use of pearson’s correlation appears sound, but seems limited, especially because this requires training a linear predictor on the output, reducing the amount of test data to 1/7 of the movie segments tested. This method needs better justification citing previous work that has established this metric, or ideally an additional metric to validate it. The PCA analysis in particular needs to be explained and justified more clearly. This is true for figure 4 but especially the analysis in figure 5 is not well explained, and the one shown appears to be an illustrative example.

We fully agree with the limitations identified by Reviewer Xyvz, we have edited the manuscript extensively to address the lack of clarity regarding the choice of models and methods and to provide more insight regarding the interpretation of the results.

Questions: Why do the authors think that ViT MAEs, despite improved performance over CNNs, do not show strong representational alignment? Some discussion of this would help strengthen the paper.

This is an excellent question. We hypothesize that this is due to a convergence between two factors. First, we found that two models explain unique neural variance compared to most other models: s-sup dynamic and sup dynamic. They are the only models trained to use optic flow information. They explain unique neural variance compared to MAEs, and also compared to other models that have analogous architecture (the static CNNs trained on the same dataset). We have added a new model to the analysis, a s-sup dynamic model trained on Kinetics (so that it is trained with the same visual diet as the Video MAEs). In line with our interpretation of the results, the Kinetics-trained s-sup dynamic model accounted for unique neural variance compared to the Video MAEs despite having been trained on the same dataset. We have summarized this key point as follows:

“As a key takeaway, the results show that the CNN models using optic flow (namely, s-sup dynamic and sup dynamic) explain unique neural variance that is not captured by MAE models (Figure 2b, columns 3-5 and rows 7-12 of the matrices). Importantly, they also explain unique variance that is not captured by other CNNs – even when they are trained with the same dataset (HAA, Figure 2b, columns 4-5, row 2 of the matrices). The results therefore indicate that the difference between CNN and Transformer architectures alone is not sufficient to account for the unique variance in neural responses explained by the models using optic flow.”

Importantly, additional factors beyond optic flow are likely to contribute to the difference in representational alignment of ViT MAEs. This is highlighted in particular by the fact that the static sup CNN trained with ImageNet shows substantially higher representational alignment with neural responses compared to ViT MAEs. In order to further examine whether this phenomenon is limited to MAEs or whether it extends to other self-supervised vision transformers as well, we have included Dyno v2 in the analysis – a Vision Transformer trained with a different self-supervised learning objective. Dyno v2 also showed generally poor alignment with neural responses, showing that the worse alignment is not restricted exclusively to Vision Transformers trained with masked autoencoding.

“It is important to note however that the representation of optic flow is likely not the only contributor to the worse alignment of the MAEs. In fact, even when trained with an action dataset (HAA), the static CNN showed better alignment with neural responses compared to the Image MAEs, and accounted for unique neural variance that was not captured by the MAEs.”

We also highlighted this in the Conclusion:

“The difference in alignment with neural responses between MAEs and CNNs is likely also driven in part by additional factors above and beyond optic flow. In particular, the comparison between ImageMAEs and the static net trained with ImageNet indicates that differences in architecture and task also play an important role for the differences in alignment with neural responses.”

Relatedly, more summarization of the results throughout the paper would make this easier to read.

We appreciate this suggestion, we have added a brief summary of the implications of each result. Note that Figures 2,3 have been combined into a single Figure (Figure 2), Figures 4 and 5 have been renamed to Figures 3 and 4. Here is a list of the edits:

“As a key takeaway, the results show that the CNN models using optic flow (namely, s-sup dynamic and sup dynamic) explain unique neural variance that is not captured by MAE models (Figure 3, columns 3-5 and rows 5-9 of the matrices). Importantly, they also explain unique variance that is not captured by other CNNs – even when they are trained with the same dataset (HAA, Figure 3, columns 3-4, row 2 of the matrices). The results therefore indicate that the difference between CNN and Transformer architectures alone is not sufficient to account for the unique variance in neural responses explained by the models using optic flow.”

and

“In other words, the results of principal component analysis (Figure 3) reveal that layers in the models using optic flow representations encode representations with a fundamentally different representational geometry compared to the other models. This is evidenced by the higher loadings of the optic flow models on the second principal component. By contrast, layers in the MAE models as well as in the CNNs that do not use optic flow information have lower loadings on the second principal component.”

As well as

“In conclusion, three models in particular explained unique variance in the neural responses that was not captured by other models: s-sup dynamic trained with Kinetics, s-sup dynamic trained with HAA and the flow classifier trained with HAA. These are the three models that represent optic flow information. The results converge to indicate that the use of optic flow information is a key factor that differentiates these models from the other models tested.”

We are working on the responses to the other comments, and we will post them shortly.

How do the authors account for differences in temporal resolution between these frame-by-frame models, which operate on videos with <100ms frame rate, and the much lower temporal resolution of fMRI?

This is an excellent question. Model representations were computed at the original frame rate (25Hz), and were then downsampled to a rate of 5Hz. This yields a temporal resolution that is still higher than that of fMRI responses. Next, features were extracted from the models’ layer at each frame. Next, the features were convolved with a standard haemodynamic response function (HRF) to account for the shape of the BOLD response. Finally, the convolved features were further downsampled to the temporal resolution of fMRI data to obtain model RDMs matching the size of the neural RDMs. We hypothesized that extracting features at a higher temporal resolution and downsampling after the convolution with the HRF would yield more accurate results by preserving information about the impact of finer grained temporal variation in visual features on the BOLD responses.

To improve clarity and ease of summarization, the authors should have more visualizations that summarize data effectively. For example, figure 1 could have an additional plot that compares all brain areas together, normalized by the noise ceiling and/or the explainability of the best model (CNN).

Because F6 is in the supplemental, F1/2/3 should label Face/Body/Artifact/Scene on the left side rows to make this designation clear.

As recommended, we have added labels to Figure 1 indicating what regions are selective for Faces/Bodies/Artifacts/Scenes. In Figures 2 and 3 (now combined into Figure 2), due to the large number of matrices we do not report the matrices of regions selective for different categories on different rows, and since preferred category does not seem to be a main driver of the effects we have omitted this information. Including a normalized version of the plots is a good idea, but since several new regions were added to Figure 1 (V1v, V1d, V2v, V2d, V3v, V3d, V4) in order to respond to other suggestions, we are short on space available to add more plots.

Give more justification for the PCA section, especially figures 4 and 5. I have read through this section and figures multiple times and am still unclear why these analyses are helpful in understanding the differences between image/video/optic flow model representations, I especially find figure 5 confusing - why this is useful to visualize in terms of improving understanding of representational alignment to human fMRI data? Was this analysis performed over multiple frames and can it be summarized?

We apologize for the lack of clarity in the original version. Figures 4 and 5 (now renamed to 3 and 4) aimed to compare the models between each other, with the goal of identifying differences between the models that explained unique variance in neural responses (s-sup dynamic and sup dynamic) and the other models. In the original version of the manuscript, the rationale and interpretation of Figures 4 and 5 (now 3 and 4) was not adequately explained. We have addressed this limitation including the following passage in the Results section to explain the interpretation of Figure 4 (now renamed as Figure 3)

“the results of principal component analysis (Figure 3) reveal that layers in the models using optic flow representations encode representations with a fundamentally different representational geometry compared to the other models. This is evidenced by the higher loadings of the optic flow models on the second principal component. By contrast, layers in the MAE models as well as in the CNNs that do not use optic flow information have lower loadings on the second principal component.”

Weaknesses: The approach is straightforward. The conceptual and technical advance is limited. The insights provided by the paper are rather limited. It is a good paper with some interesting results, but probably on par with the standard of ICLR papers.

We appreciate the Reviewer’s evaluation that the paper is on par with the standard of ICLR papers. In order to improve the paper in the direction indicated by the Reviewer’s questions, we have expanded the analyses, clarified their rationale, and discussed in more detail the interpretation of the results.

Questions: It would be worthwhile to dig deeper to understand WHY some models are better than others and why some models are complementary.

We agree with Reviewer Tz3t that a central question concerns why some of the models – and specifically the s-sup dynamic and the sup dynamic models – explain unique neural variance that is not explained by the other models. One possible hypothesis is that these models outperform the other models due to their CNN architecture, that differentiates them from the MAE models which are based on a Transformer architecture. However, if this were the case, the sup static model trained with the HAA dataset should perform equally well (because it is also based on a CNN architecture). However, this is not the case. This suggests that the use of CNN architecture is not sufficient to explain why the s-sup dynamic and the sup dynamic models explain unique neural variance.

Another possibility is that the difference might be driven by the visual diet: the s-sup dynamic and the sup dynamic models were trained with the HAA dataset, while other models (e.g. the video MAEs) were trained using the Kinetics dataset. To rule this out, we added new analyses with a s-sup dynamic model trained with Kinetics. The new analyses show that even when trained on the same dataset as the Video MAEs (Kinetics), the s-sup model still explains unique neural variance that is not accounted for by the Video MAEs, suggesting that differences in the visual diet are not sufficient to explain why the s-sup dynamic and sup dynamic models explain unique neural variance that is not accounted for by the Video MAEs.

Finally, an alternative hypothesis is that the s-sup dynamic and the sup dynamic models differ from all other models in terms of their representation of optic flow information. Figure 4 shows that these models differ from all other models along the second principal component in representational space. However, Figure 4 alone does not tell us whether this second principal component is related to optic flow information. For this reason, we performed the analysis shown in Figure 5. We identified pairs of timepoints in the video that show large differences along principal component 2, and compared them. The comparison reveals that timepoints that differ along principal component 2 are very different in terms of the optic flow present at those timepoints. In other words, this result indicates that principal component 2 is associated with differences in optic flow, and thus ultimately that the models that account for unique neural variance (s-sup dynamic and sup dynamic) differ from the rest of the models in terms of their optic flow representations.

We have edited the manuscript to explain these results more clearly, adding the following passages to the Results section:

“As a key takeaway, the results show that the CNN models using optic flow (namely, s-sup dynamic and sup dynamic) explain unique neural variance that is not captured by MAE models (Figure 3, columns 3-5 and rows 5-9 of the matrices). Importantly, they also explain unique variance that is not captured by other CNNs – even when they are trained with the same dataset (HAA, Figure 3, columns 3-4, row 2 of the matrices). The results therefore indicate that the difference between CNN and Transformer architectures alone is not sufficient to account for the unique variance in neural responses explained by the models using optic flow.”

And

“Importantly, the self-supervised dynamic model accounts for unique neural variance compared to the Video MAEs even when trained on the same dataset: Kinetics (Figure 3 matrices, column 3, rows 10-12). This indicates that the difference in performance between s-sup models and Video MAEs cannot be fully attributed to differences in the visual diet.”

Weaknesses: I think the analysis done in this paper do not correspond to the papers' main conclusion. For example, the conclusion at the end of the paper that states that computer vision dynamic models are more neurally aligned with brain data vs static models is not really supported in Figure 4 or Figure 5. In Figure 4, the RDM brain trajectories are of several models plotted compared against each other. I would have excepted to see a line as well for the "visual cortex" all-together (human ground truth, and incrementally from V1,V2,V4, IT etc...) so that we can qualitatively make an assessment of human vs machine alignment. Further figure 5 also seems strange: What does the 1st and 2nd PC across frames have to do with saying that dynamic models are better than static ones. It overall feels like the RDM analysis is done incorrectly. I am under the impression that the figures I would be ideally looking at is comparing human vs machine feature outputs or or recording for a collection of visual stimuli. Instead, models are being compared to each other given their activation per brain region. So it somehow feels like the analysis was being done within vs between systems.

We appreciate Reviewer mTch’s constructive feedback, and we realize that our writing in the first version of the manuscript lacked clarity. When Reviewer mTch notes “I am under the impression that the figures I would be ideally looking at is comparing human vs machine feature outputs or or recording for a collection of visual stimuli.”, they are indeed correct. The figures comparing human recordings to machine outputs are Figures 1, 2 and 3 (now consolidated into Figures 1 and 2).

The manuscript is organized into two sets of analyses. The first set of analyses, corresponding to Figures 1, 2 and 3 (now 1 and 2), compares the computer vision models to brain data. The remaining analyses investigate more closely the differences between the models that explain unique variance in neural responses (s-sup dynamic and sup dynamic) and all other models. Former Figures 4 and 5 (now 3 and 4) indeed do not contain comparisons with brain data, because those comparisons have been completed in the analyses shown in the previous Figures.

We have added a section to the Methods explaining more clearly the brain-model comparisons, titled “Comparison between models and neural responses”. The section includes the following explanation of the brain-model comparisons:

“Models were compared to neural responses using Representational Dissimilarity Matrices (RDMs, \cite{kriegeskorte2008representational}). In this study, RDMs are matrices whose rows and columns correspond to timepoints in the movie, such that the element of the matrix at a given row and column is the dissimilarity between the representation of the video at the timepoints that correspond to that row and column. Neural RDMs and model RDMs were compared by computing their Pearson correlation.

The match between neural RDMs and RDMs for an entire model were calculated by first computing RDMs for each layer of the model and then computing a linear combination of the layer RDMs that best matches the neural RDM. In order to prevent circularity in the analysis, the weights attributed to each layer in the linear combination were calculated using 7 of the 8 experimental runs, and were applied to the model RDMs in the left-out run to compute a “predicted” RDM. We then evaluated the correlation between the “predicted” RDM and the neural RDM in the corresponding run (Figure 1).”

The goal of the analysis in Figure 2 (which is now Figure 2a after the revision) was to determine whether combining multiple models provides a better account of neural responses compared to using a single model. We added this explanation to the Methods section:

“To evaluate the joint neural predictivity for each pair of models, we computed a linear combination of the layer RDMs from both models that best matches neural responses. The weights for each layer were calculated using 7 of the 8 experimental runs, and the correspondence between the models and neural responses was then evaluated on the left-out run. This analysis enabled us to test whether and to which extent using two models jointly yielded a better match to neural responses than using a single model.”

And regarding Figure 3 (which is now Figure 2b after the revision):

“Finally, we wanted to test more directly the unique variance in a neural RDM that was explained by a model above and beyond each other model. To compute this, we regressed out a control model RDM from a neural RDM, and predicted the residual neural RDM with a target model, obtaining the unique variance explained by the target model. Matrices in Figure 2b show these difference values, with the target models as the columns and the control models as the rows.”

We also clarified the key take home message in the Conclusion:

“Convolutional models based on optic flow explained unique variance in neural responses that was not accounted for by any other model, not even video MAEs. Analysis of the representational geometry in the different layers of the models revealed that the second principal component in the space of representational dissimilarity matrices (RDMs) distinguished between convolutional models based on optic flow on one hand (which scored highly on the component) and all the other models on the other hand, suggesting a critical role of optic flow representations in human vision. We probed this conclusion further by examining the loadings of this component, and identifying pairs of scenes in the movie that were differentiated by the models based on optic flow but not by the other models. These included scenes with similar entities and backgrounds, that differed in the presence or absence of overall background flow (e.g. due to movement of the camera), further supporting the conclusion that video MAEs do not encode a set of dynamic features that are instead computed by both optic flow models and by human vision.”

And

“The difference in alignment with neural responses between MAEs and CNNs is likely also driven in part by additional factors above and beyond optic flow. In particular, the comparison between ImageMAEs and the static net trained with ImageNet indicates that differences in architecture and task also play an important role for the differences in alignment with neural responses.”

And

“In conclusion, the results converge to indicate that the lack of optic flow representations and the use of self-supervised Vision Transformer architectures are jointly responsible to account for decreased alignment between models and neural representations.”

Questions: Why video CNNs were trained on a different dataset as compared to video MAEs

To address this limitation, we have additionally trained a video CNN using Kinetics, the same dataset used to train Video MAEs. The video CNN trained with Kinetics yielded a pattern of results analogous to the video CNN trained on HAA, indicating that these results are not due to differences in the training dataset.

How do authors infer that different types of dynamic information are represented in OFA (face-selective), EBA (body-selective), and TOS (scene-selective). Line 367

We meant to say that dynamic information contributes more to accounting for EBA and TOS responses, this is based on the higher values in Figure 3, columns 3-4 for those regions. We have amended the text accordingly.

We thank Reviewer Wd8d for the positive evaluation of the manuscript, and report our responses below.

Weaknesses: Results are reported on a single fMRI dataset. With large scale public fMRI datasets e.g. NSD, Algonauts videos available, the authors could have reported results on multiple dataset and showed generalizability of their results

We agree with Reviewer Wd8d about the importance of the generalizability and robustness of the results. Due to the time constraints for resubmission we were not able to redo the analyses on an entirely new dataset. One aspect we would like to note, however, is that the prediction of neural RDMs was computed by estimating weights with 7 of the 8 experimental runs, and by evaluating the brain-model correlation using data from the left-out run. While this does not fully address the Reviewer’s comment, we believe that due to the differences between different sections of the movie it helps to support the generalizability of the results.

I am not sure how different are individual conditions in the RDMs. If the conditions are part of the same movie then there could be high correlation between the individual conditions

In the RDMs, each row/column corresponds to a different timepoint in the movie. Some pairs of timepoints are very similar, while others are very different, depending on the similarity between the scenes depicted at those different timepoints. The movie includes a wide variety of different scenes, that include indoor and outdoor scenes as well as scenes with a low and high amount of movement, leading to a broad range of correlation values.

Lack of interpretations of the results. While the authors shows model X show less correlation than model Y. An interpretation of why this could be happening is missing in the text and its relevance to brain regions functions

We concur with Reviewer Wd8d that the original version of the manuscript did not adequately discuss the interpretation of the results. To address this limitation, we have made the following additions to the Results and Discussion sections of the article (Figures 2,3 were consolidated to Figure 2, and Figures 4,5 renamed to Figures 3,4):

“As a key takeaway, the results show that the CNN models using optic flow (namely, s-sup dynamic and sup dynamic) explain unique neural variance that is not captured by MAE models (Figure 2b, columns 3-5 and rows 7-12 of the matrices). Importantly, they also explain unique variance that is not captured by other CNNs – even when they are trained with the same dataset (HAA, Figure 2b, columns 4-5, row 2 of the matrices). The results therefore indicate that the difference between CNN and Transformer architectures alone is not sufficient to account for the unique variance in neural responses explained by the models using optic flow.”

“The additional unique variance explained by the optic flow models (s-sup dynamic and sup dynamic) varied across regions, being strongest in EBA and TOS and weakest in FFA. The effect was observed widely, in regions previously associated with the processing of dynamic information (such as STS), but also in ventral temporal regions that have not been typically associated with the representation of dynamics (such as PPA). This observation is consistent with recent work suggesting that dynamic information is represented in a broader range of brain regions than previously thought (\cite{robert2023disentangling}).”

“In other words, the results of principal component analysis (Figure 3) reveal that layers in the models using optic flow representations encode representations with a fundamentally different representational geometry compared to the other models. This is evidenced by the higher loadings of the optic flow models on the second principal component. By contrast, layers in the MAE models as well as in the CNNs that do not use optic flow information have lower loadings on the second principal component.”

“It is important to note however that the representation of optic flow is likely not the only contributor to the worse alignment of the MAEs. In fact, even when trained with an action dataset (HAA), the static CNN showed better alignment with neural responses compared to the Image MAEs, and accounted for unique neural variance that was not captured by the MAEs.” “In conclusion, three models in particular explained unique variance in the neural responses that was not captured by other models: s-sup dynamic trained with Kinetics, s-sup dynamic trained with HAA and sup dynamic trained with HAA. These are the three models that represent optic flow information. The results converge to indicate that the use of optic flow information is a key factor that differentiates these models from the other models tested.”

I appreciate the authors' responses. However, I do not think the manuscript is ready for publication. I would maintain my score to be 5 and improvements can be added for the next version. Moreover, I do not see the updated version of the manuscript, have you updated it or just prepared to update?

1. Response to: "The scientific hypothesis we reach from this study is that the misalignment between MAE models and human brains might be due to how the human brain encodes information about optic flow."

I am afraid that this hypothesis is already falsified by some results in the main text. Referring to Fig.2 FFA, a sole supervised CNN can predict responses in this area and including other models do not improve the predictivity. Moving on to MAE, it cannot predict FFA responses well. This is not because of optical flow because CNN that predict the responses well with or without including optical flow representation from other CNN models. MAE has low predictivty for all regions regardless of the need for optical flow representation to better align with those regions.

2. This demonstrates that the CNN architecture on its own is not sufficient to account for the unique variance in neural responses, and that the improved correspondence with neural responses is restricted to the models that use optic flow (s-sup dynamic and sup dynamic).

CNN or whatever architectures trained to predict optical flow would better explain responses in some brain areas that estimate the optical flow. I would not be surprised because CNN (sup dynamic) is trained to predict what some brain areas are doing. For some areas like FFA, there is no need for optical flows. Moreover, why is this related to the main hypothesis about MAE?

3. Response to explanation of Fig 2 and 3.

In my opinion, Fig 2 and Fig 3 are highly dependent. Having only one of those figures would be enough to explain the "unique representations of the neural networks". I would like to know more about the cases where we need both Fig 2 and Fig 3 to explain the model. Correct me if I am wrong.

For Fig 2, each element in the matrix comes from the "neural predictivity score" of combining two network models' responses in each row and column. Y = f(A,B) where f is the regression model and A and B are row and column model RDM and Y is the neural RDM.

For Fig 3, each element comes from the "residual predictivity". To get the residual, the first model (row) predicts neural RDM and then residuals can be computed. Residual is then predicted by the second model (column) and the predictivity score can be computed based on how accurate the target model can predict the residual.

Essentially, Fig2 and Fig3 matrices are from combining two models' responses (model RDM) to predict neural responses (neural RDM).

Weaknesses: The conclusion of the paper does not lead to any scientific theory or a glimpse of it. Specifically, it is unclear whether the misalignment between MAE and human brains is due to Transformer Architecture or the Masked Autoencoder pre-training itself. The study has to be more concrete to conclude the theory. To decouple the confounders, I suggest the authors do more experiments on Masked Autoencoder with CNN and see if the low neural alignment still exists. The authors may also play around with different “noises” as Masked Autoencoder is a special case of Denoising Autoencoder. If Masked CNN is much more aligned with the brain, it is probably because of Transformer architecture that causes misalignment. Last but not least, the authors may use other kinds of pre-trained vision transformers that work equally well compared to MAE pre-training, such as MoCo v3, Dino v2. If Transformer is really misaligned with the brain, I would expect a low alignment regardless of the pre-training method. If it is because of the Masked Autoencoder, I would expect high alignment.

We thank Reviewer kFNo for this feedback, we agree that we could have been clearer about the scientific conclusions that can be obtained from the study. The scientific hypothesis we reach from this study is that the misalignment between MAE models and human brains might be due to how the human brain encodes information about optic flow. We discuss the details of our logic below. We have updated the Figures with additional analyses, the response refers to the updated Figures. Specifically, we have added results for early visual regions (V1-V4) in Figure 1 as Figure 1b, and we have combined the former Figures 2 and 3 into Figure 2 (as 2a and 2b).

First, we observe that CNN models trained to learn optic flow (s-sup dynamic) as well as CNN models that use optic flow as input (sup dynamic) account for unique variance in neural responses that is not captured by the MAE models (see matrices in Figure 2b, columns 3 to 5 from the left, rows 6 to 12 from top). Importantly, CNN models that use optic flow information also account for unique variance in neural responses that is not captured by other CNN models (see matrices in Figure 2b, columns 3 to 5 from the left, rows 1 and 2 from top). This demonstrates that the CNN architecture on its own is not sufficient to account for the unique variance in neural responses, and that the improved correspondence with neural responses is restricted to the models that use optic flow (s-sup dynamic and sup dynamic).

To investigate this phenomenon in more detail, we treated the representational dissimilarity matrix of each model’s layer as a high-dimensional vector. Therefore, analyzing all models and layers, we obtained a set of vectors, and performed principal component analysis (PCA) to identify dimensions that account for most variation in the representations encoded in different layers of different models (Figure 3). In line with the observations noted in the previous paragraph, we found that the models that use optic flow learn very different representational structure compared to both MAE models and other CNNs (see the green, red and purple lines). This observation converges with the argument outlined in the previous paragraph to indicate that the differences are not due to an overall distinction between CNN and Transformer architectures. Instead, among CNNs, those that use optic flow learn different representations compared to all other models, including video MAEs.

At this point, we wanted to obtain a more interpretable understanding of the stimulus features driving the differences between the models that use optic flow and the rest of the models. We noted that the models that use optic flow mostly differ from the rest of the model along the second principal component (PC2). Principal components have loadings on each element of the representational dissimilarity matrix. Since the representational dissimilarity matrix contains the dissimilarity between different timepoints in the video, we could identify which pairs of timepoints in the video have high loadings for PC2. We visualized 8 example pairs of timepoints with high loadings for PC2 in Figure 3 (formerly Figure 4), showing the frames as well as optic flow maps at those timepoints. Typically, one timepoint in the pair is characterized by more optic flow than the other, further supporting the hypothesis that the differences observed between the models are driven by how they represent optic flow information.