Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream

Thank you for your comprehensive and thoughtful review of our manuscript. Below, we address each of your questions in detail.

1. Could there be additional context on the novelty of this work relative to existing literature on model size effects?

Thank you for emphasizing the importance of contextualizing our work within the existing literature on model size effects. Previous work primarily showed scaling with respect to ground-truth performance. Our work evaluates scaling behavior of models with respect to their internal similarity to brain responses, which has to date not been attempted as far as we are aware. Our study introduces several novel contributions that set it apart from previous research:

1. Comprehensive Model and Dataset Exploration: While previous studies have examined specific aspects of model scaling, our work systematically explores over 600 models across various architectures and dataset sizes. This extensive evaluation provides a more holistic understanding of how scaling dimensions interact to influence neural and behavioral alignment.
2. Differential Scaling Laws for Neural and Behavioral Alignment: Our research identifies distinct scaling laws for neural and behavioral alignment, revealing that while behavioral alignment continues to improve with scale, neural alignment reaches a saturation point. This differentiation offers deeper insights into the limitations of current scaling strategies and their differential impacts on various alignment metrics.
3. Scaling Recipe for Optimal Compute Allocation: We introduce a novel scaling recipe that optimally allocates compute between model size and dataset size to maximize alignment (Fig 4). This practical guideline is a significant advancement, offering actionable recommendations for future model training strategies aimed at enhancing brain alignment.
4. Granular Analysis Across VVS Hierarchy: Our study delves into how scaling impacts different regions within the primate visual ventral stream (VVS), from V1 to IT and behavioral outputs (Fig 5). This hierarchical analysis reveals that higher-level regions benefit more from scaling, a detail that had not been thoroughly examined in prior work.
5. Public Release of Extensive Resources: By open-sourcing our training code, evaluation pipeline, and a vast collection of model checkpoints, we provide invaluable resources for the research community. This transparency facilitates reproducibility and enables other researchers to build upon our findings, thereby accelerating progress in the field.

These contributions collectively advance the understanding of how model scaling influences alignment with both neural and behavioral aspects of the primate visual system, offering new perspectives and practical tools that were not previously available.

2. Is it possible to control inductive biases more rigorously, either quantitatively or qualitatively?

We have expanded the investigation of inductive biases and their impact on alignment in the updated manuscript. Specifically, we analyze the evolution of neural and behavioral alignment during supervised training across different architectures (Figure 8c). Our results confirm that models with strong priors, such as convolutional architectures, exhibit higher neural alignment at initialization compared to more generalist models, like vision transformers.

Additionally, we present a detailed comparison of alignment at initialization across architectures in Figure 11 (Appendix), further supporting the role of inductive biases in early alignment.

We also explore how inductive biases interact with different training objectives in Figure 8d. For example, while the alignment of a ResNet model shows only slight variation between supervised and self-supervised objectives, the alignment of a ViT model is significantly influenced by the training objective. Notably, the self-supervised objective provides a richer learning signal, resulting in a faster rise in alignment during training. This suggests that inductive biases, combined with learning objectives, play a critical role in shaping alignment dynamics.

We sincerely appreciate your review and the time you invested in evaluating our manuscript. We strongly feel that this submission is a significant contribution to the NeuroAI field and that it should thus be featured at ICLR. We address your concerns and questions below.

1. Novelty and Contribution to the Field

Concern: You expressed concern about the novelty of our findings and how our work contributes to the field, noting similarities with prior studies such as Linsley et al.

Response: Previous studies have indeed explored the relationship between task performance and brain alignment. Almost all of them found a continued positive relationship. We are only aware of two papers (Schrimpf et al. 2018 and Linsley et al.) that raised the point this relationship might break for data from a single visual area (IT). Our work provides substantially novel findings in several key aspects:

• Dissociation Between Neural and Behavioral Alignment: Our findings highlight a clear dissociation between neural and behavioral alignment as models scale (Figs. 1b, 5, 7a, 13). While behavioral alignment continues to improve, neural alignment saturates—a phenomenon not quantitatively characterized in prior work.
• Unexpected saturation: Across many domains of brain function, larger and more task-performant models lead to improved alignment with brain data (e.g. vision $Yamins et al. 2014$ , auditory $Kell et al. 2018$ , language $Schrimpf et al. 2021$ , motor $Vargas et al. 2024$ ). It is thus reasonable to believe that continued performance scaling will yield continued brain alignment gains, and in our experience this is the reality for most of the field; for instance virtually all models on Brain-Score are pre-trained machine learning models. We show that scaling the ML way will not improve alignment to the brain’s visual system, and pinpoint the primary failure cases to early visual processing (see below).
• Graded Effect Across Brain Regions: We uncover an ordered effect of scaling on alignment across different brain regions in the visual hierarchy (V1, V2, V4, IT; Figs 5, 13), providing insights into how scaling differentially impacts various levels of neural processing.
• Systematic Quantification: We provide a systematic and controlled investigation into how scaling both model size and dataset size affects neural and behavioral alignment. By training over 600 models under controlled conditions, we eliminate confounding factors present in studies using pre-trained models with varying architectures and training regimes.
• Parametric Scaling Laws: We introduce and fit parametric power-law scaling laws to our data, offering a predictive framework for how alignment scales with compute and data. This quantitative approach allows us to extrapolate and predict alignment at scales beyond those directly tested. Indeed, we further validated these predictions with unsupervised, multimodal, and adversarially trained variants (new Figs 7ab, 9, and 13).
• The Role of Inductive Biases: Our analysis demonstrates how inductive biases in neural network architectures (e.g., convolutions in ResNets vs. transformers) affect alignment. In the revised manuscript, we present training dynamics (Figs. 2, 7c, 10, 11 ) showcasing how models with differing biases converge to similar representations over time, albeit starting from distinct initial points. This sheds light on the influence of architectural priors on neural and behavioral alignment.
• Direction Forward: Our study provides quantitative scaling laws that highlight how computational resources can be effectively allocated between model size and dataset size to optimize neural and behavioral alignment. While these findings underscore the benefits of scaling especially for behavioral alignment (a positive finding), the observed saturation in neural alignment suggests that scaling alone is insufficient to achieve more accurate models of the primate visual ventral stream.
  We discuss the necessity of exploring alternative strategies in Discussion Limitations and Future Directions section, such as integrating biologically inspired architectural features (e.g., V1 block in VOneNet), co-training with brain data, and developing novel training objectives tailored to better capture neural dynamics. Additionally, our experiments with adversarial fine-tuning demonstrate its potential in raising alignment saturation levels, suggesting that robust training approaches could play a crucial role. As such, combining stronger inductive priors with advanced training paradigms like adversarial fine-tuning offers a promising path toward developing next-generation models that more faithfully mimic biological vision systems.

We believe these contributions offer new insights and a valuable direction for future research, emphasizing that scaling alone may not suffice to improve neural alignment, thereby highlighting the need for novel modeling approaches.

1. Lacking evaluation of what model behaviors give rise to alignment:

We agree that understanding the specific model behaviors that enhance alignment is crucial for advancing the field. While our current study focused on quantifying scaling laws, we are actively investigating the qualitative factors that contribute to neural and behavioral alignment. One aspect we are exploring is the sensitivity of models to spatial frequencies and other visual features that are relevant to human perception. Additionally, we are analyzing the eigenspectrum of response characteristics of these models to identify patterns that correlate with improved alignment.

2. Evaluation of Recent Multimodal Models:

We acknowledge the growing importance of multimodal vision-language models in current AI research. We have conducted additional evaluations on publicly available large vision-language models, such as CLIP and DINOv2. These models include much larger architectures and are trained on extensive datasets, including LAION, offering a broader scope of training data and objectives compared to our controlled experiments. Our findings reveal that while these multimodal models achieve enhanced behavioral alignment—likely due to their diverse training objectives and data sources—their neural alignment still exhibits a saturation effect similar to that observed in unimodal models. This indicates that, despite the advanced training paradigms and data diversity, multimodal models do not fundamentally address the limitations in neural alignment scaling. We have incorporated these results into the revised appendix, including a detailed visualization of the scaling behavior of pretrained models (Figure 9). When compared to Figure 2c, which illustrates the scaling behavior of models we trained, the similar saturation levels observed between pretrained and trained models reinforce the generalizability of our findings. These results provide additional evidence that while scaling improves behavioral alignment, neural alignment requires alternative approaches to overcome the observed limitations.

Thank you for your insightful and constructive review of our manuscript. We are pleased that you found our work well-written and that you recognize the novelty in demonstrating varied scaling effects across different cortical areas. Below, we address each of your comments and questions in detail.

1. What are the implications of this work, given the limitations already presented in the paper?

Given the limitations presented in the paper—such as the specific range of model sizes and dataset volumes examined, the subset of architectures evaluated, and the datasets used—the key implications are as follows:

1. Scaling Alone Is Insufficient for Neural Alignment: The study reveals that while scaling up model parameters and training data consistently enhances behavioral alignment with human performance, it leads to saturation in neural alignment with the primate visual ventral stream (Figs 1b, 5, 7a). This indicates that simply increasing scale using current architectures and datasets is not enough to achieve better neural alignment (Fig 9). The implication is that alternative approaches are necessary to develop models that more accurately mimic neural representations in the brain.
2. Need for Alternative Modeling Approaches: The observed saturation in neural alignment suggests that future research should explore new strategies beyond traditional scaling (Fig 2, 7a). This includes integrating biologically inspired architectural features such as feedback mechanisms, leveraging additional data modalities, and developing novel training objectives tailored to better capture the dynamics of neural processing.
3. Importance of Inductive Biases: The findings highlight the significant role of architectural inductive biases in achieving neural alignment. Models with strong inductive biases, like fully convolutional networks (e.g., ResNets and EfficientNets), demonstrate higher initial neural alignment even before training (Figs 2, 7c, 10, 11). This implies that incorporating architectural priors that reflect biological neural structures can improve alignment efficiency without solely relying on scaling.
4. Guidance for Resource Allocation: By introducing and fitting parametric power-law scaling laws, the study provides a predictive framework for how alignment scales with compute and data (Fig 4a). This quantitative approach offers practical guidance on how to allocate computational resources effectively between model complexity and dataset size to optimize both neural and behavioral alignment.
5. Potential of Adversarial Training and Alternative Learning Signals: The experiments with adversarial fine-tuning show its potential in enhancing neural alignment beyond the saturation levels observed with standard training methods (Fig 7b). This suggests that incorporating robust training approaches and alternative learning signals could play a crucial role in developing models that better align with neural data.
6. Differential Impact Across Brain Regions: The discovery of an ordered effect of scaling on alignment across different brain regions provides deeper insights into how scaling differentially affects various levels of neural processing (Fig 5). This suggests that scaling strategies may need to be tailored to target specific regions within the visual cortex to achieve optimal alignment.

In summary, the implications of this work emphasize that while scaling is beneficial for improving behavioral alignment, it is not sufficient for advancing neural alignment with the brain's visual system using current models and datasets. This underscores the necessity of exploring new modeling approaches that incorporate biological principles and alternative training strategies. Despite the limitations, such as the specific models and datasets used, these findings offer valuable insights and directions for future research aimed at bridging the gap between artificial neural networks and biological neural processing.