Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream

Thank you for your thoughtful review — we’re glad you found the paper well-written and organized, appreciated the breadth of our experiments and visualizations, and found the findings insightful. We respond to each of your comments below point-by-point.

The practical implications of the authors' conclusions remain unclear. For instance, while the authors highlight that models with strong architectural inductive biases, such as convolution-based ResNets, show better neural alignment trends than transformer-based ViTs, ViTs are still more widely adopted in the community due to their superior overall performance compared to convolution nets.

Since our primary goal is modeling brain information processing (i.e., neuroscience), we consider task performance mainly as a proxy rather than the ultimate target. Indeed, our experiments show ViTs excel at behavioral alignment (Figs. 5a, 6b) but underperform relative to convolutional models in neural alignment benchmarks (Figs. 5a, 6a). This also holds for very large models trained on massive datasets (Fig S2, ConvNeXt vs ViT). Practically, this suggests a promising path forward could involve hybrid architectures that combine biological inductive biases (e.g., convolutional or hierarchical connectivity) with transformer-based scalability and flexibility such as a VOneBlock (Dapello et al. 2020) combined with ViTs (Dosovitskiy et al. 2020), thus balancing task performance with improved neural alignment. Specifically, one viable strategy could be distilling representations from high-performing transformer models into architectures with hierarchical inductive biases resembling biological circuitry.

The authors reveal the scaling laws between neural alignment, behavioral alignment, and data size, model size through a series of impressive experiments. However, what is the relationship between improving neural alignment and enhancing downstream performance? What insights can these scaling laws provide for improving model design?

In this study, we primarily focus on building better models of the human brain, rather than developing better ML models for various tasks. In this context, our findings show a clear correlation between behavioral alignment and task performance (Figs. 1, 6b), consistent with the widely accepted notion in the ML community that increased compute leads to better-performing models. In contrast, neural alignment shows positive but diminishing returns with improved object recognition performance (Fig. 6a), indicating that improving neural alignment may not always directly translate into better downstream task performance.

Practically, our scaling laws offer concrete guidance for future model design targeting neural alignment. First, our results strongly suggest prioritizing larger, richer, and more diverse datasets, as they consistently yield greater neural alignment improvements compared to merely scaling model complexity (Fig. 3a). Second, our findings emphasize the crucial role of biologically inspired inductive biases, particularly in scenarios constrained by limited compute or data, as these priors substantially enhance alignment efficiency (Figs. 2, 3b). Lastly, the graded scaling effect across the cortical hierarchy (Fig. 5) indicates that early visual regions (V1/V2) are the most challenging to align through scale alone, suggesting that architectures incorporating stronger biologically-informed priors (e.g., VOneNet-style architectures) may be necessary to achieve further improvements.

Thanks again for your review. We ask the reviewer to consider raising their score in light of this paper’s placement in computational neuroscience and a focus on modeling the brain.

We thank the reviewer for their positive feedback and thoughtful comments. We are particularly encouraged that the reviewer highlighted the key strengths that we were indeed most excited about in our paper:

• Our systematic, extensive evaluation of scaling laws across hundreds of models and diverse benchmarks, providing a new insight into the role of architectural bias and choice of training, and a recipe for optimal compute allocation.
• Thorough comparisons across various architectures (ResNet, EfficientNet, ViT, ConvNeXt, CORnet), multiple datasets (ImageNet, EcoSet, iNaturalist, infiMNIST, Places365), and alternative training methods (SimCLR, DINO, adversarial training).
• Clear evidence showing neural alignment saturates with increased model and dataset scaling, providing a significant conceptual step forward to encourage the exploration of new modeling ideas.
• The depth and rigor of our analyses, including validation with held-out benchmarks and alignment dynamics during training, showing for instance that under controlled model training, task performance and brain alignment remain weakly correlated.
• Comprehensive results specifically targeted at the primate visual cortex, offering deeper insights beyond prior work, such as the ordered effect of scale on alignment and the role of inductive biases.

We greatly appreciate the reviewer's supportive assessment and enthusiasm for our study.

We appreciate your positive review and are glad you found the paper well-written, the results clear, and the evidence supporting our claims solid. Below, we address your questions point by point.

In light of these previous works, it might be expected that scaling up models and data would be insufficient to achieve better alignment for neural data. Therefore, the main findings of the paper are not all that surprising and I find the novelty of the paper to be limited. The findings are empirical in nature and do not provide a deeper understanding for the reasons behind the observed scaling laws, other than the inductive biases of the models.

Our work substantially advances previous literature by providing a more systematic, controlled, and comprehensive analysis that isolates how individual factors—model parameters, dataset size, and compute resources—each influence alignment with multiple brain regions and behavior. Earlier studies on the other hand used heterogeneously pre-trained models and compared to limited brain datasets.

• Generalization of brain regions: Prior works focused narrowly (e.g., only IT in Linsley et al., 2023) or used noisy fMRI signals (Conwell et al., 2022). We evaluate alignment across the full primate ventral stream (V1–IT) using high-resolution intracortical recordings, along with object recognition behavior—both novel in this context.

• Model comparability: Previous studies used pretrained models with inconsistent training recipes, making comparisons hard. Our models are all trained from scratch under uniform conditions, enabling a fair, apples-to-apples comparison across architectures and datasets.

• Our comprehensive study led to updated findings: We show that task performance does correlate with neural alignment (Fig. 6a), but with diminishing returns—contrary to prior conclusions. Moreover, our work quantifies the individual effects of data, parameters, and compute (Figs. 2–4), and provides a parametric framework for understanding scaling (Figs. 4–7), going well beyond simple correlations in earlier work.

• Finally, we contribute new insights such as a graded scaling effect across the visual hierarchy and a clear behavioral/neural alignment dissociation (Fig. 5), offering deeper understanding of how brain-like representations emerge. We’ll emphasize these distinctions more clearly in the revised manuscript and welcome suggestions for where to further highlight them.

Figure panel 3b is not described in the text.

We thank the reviewer for pointing out this oversight. Figure 3b illustrates that model families with weaker inductive biases (e.g., ViT and ConvNeXt) begin training with lower neural alignment scores and consequently require larger training datasets to achieve alignment comparable to architectures with stronger inductive biases (e.g., ResNet, EfficientNet). Additionally, we find that recurrence, as implemented in CORnet architectures, provides a significant initial advantage in alignment relative to purely convolutional models, particularly in low-data regimes. However, this advantage diminishes with increased training data. Overall, these findings emphasize that strong inductive biases—such as convolution and recurrence—facilitate better alignment when data is limited, whereas extensive task-driven optimization eventually mitigates differences across architectures. We will integrate this explanation into the main text of the revised manuscript.

In line 313, right column, you speculate that factors other than task performance influence neural alignment. Do you have any ideas about what these factors might be?

Our findings suggest that architectural inductive biases significantly influence neural alignment independently from task performance. Additionally, as illustrated in Figure 5b, higher cortical regions and behavioral alignment benefit more from task optimization compared to early visual areas. Moreover, supplementary analysis (Figure S7) shows that correlations between alignment and task performance also follow the cortical hierarchy. These observations imply that incorporating stronger biologically-inspired priors (e.g., convolutional constraints, local receptive fields, recurrence) specifically for modeling early visual regions, combined with more flexible, data-driven layers for higher cortical areas—akin to the design philosophy of architectures like VOneNet—may yield improved neural alignment. Furthermore, explicitly incorporating neural data into training procedures (via co-training or fine-tuning, see our response to the reviewer weze) represents a promising additional strategy to surpass the current limitations of purely task-optimized models.

Thanks again for your review. In light of the systematic advances over prior work—such as our controlled experimental setup, broader neural benchmarks, and novel findings on graded scaling and behavioral dissociation—we kindly ask you to consider raising your score.

Thank you for your review - we’re glad you found the paper informative and comprehensive in its overview of models and progress on neural and behavioral benchmarks. We respond to each of your comments below.

Claims And Evidence - limited model set:

The reviewer might have missed this in the paper, but a core focus of our work is to evaluate a variety of architectures (Figs 2, 3, S2) from standard CNNs (ResNet, AlexNet, EfficientNet) to recent ones (ConvNeXt), Vision Transformers (ViT, DaVit, FastViT, LeViT, MaxxViT, MobileViT) and CORnet. The recurrent architecture CORnet is notably present in all our analyses (lines 133-145 left column; Figs 3, 4, 6) but we do not observe a difference in its alignment at larger scales. Interestingly in Fig 3b, CORnet initially exhibits improved alignment in low-data conditions, indicating that recurrence might provide a meaningful inductive bias for efficiency under limited training samples. However, as the amount of training data increases, the advantage offered by recurrence diminishes, suggesting that deeper feedforward models can approximate recurrent processing by effectively "unrolling" recurrent computations through additional layers. Thus, recurrence remains beneficial primarily in data-constrained scenarios, providing sample efficiency advantages, but larger datasets reduce its relative advantage. Inspired by the reviewer’s question, we will highlight this point more prominently in the updated paper.

The paper offers a wide range of experiments that validate the scaling laws, however the results described seem to resemble facts that were reported before.

Please see our first response to the reviewer 6Eek.

Weaknesses

Please see the answer below for your first question.

How can the scaling laws inform future selection of architectures data to improve our understanding of the visual cortex?

Our scaling laws explicitly guide future architecture and dataset development across varying computational budgets. Specifically, our results suggest that architectures with biologically-inspired inductive biases—like convolution or recurrence—achieve superior alignment more efficiently, especially under limited compute (Figs 2, 3b, 4b). Furthermore, our findings strongly advocate for developing richer and more ecologically valid datasets, which consistently yield greater alignment improvements compared to increasing model complexity alone (Fig 3a, Sec 3.3). Additionally, the graded effect of scaling observed across the cortical hierarchy (Fig. 5) suggests that early visual areas (V1/V2) benefit relatively little from scaling alone, highlighting the need for stronger biologically-informed inductive biases early in the processing pathway, e.g. VOneNet. Finally, we hypothesize that to push neural encoding models beyond the current alignment plateau, explicitly incorporating neural recordings into model training may be necessary. Our preliminary results on fine-tuning models on IT neural data (Papale et al., 2024) are promising: https://imgur.com/a/okfmRu8 (anonymous link for figures). Fine-tuning on a single region from another dataset improves neural alignment, with a stronger effect for a weakly-biased model (ViT-Small) than for a CNN (ResNet50). We will clearly discuss these directions and practical implications in our revised manuscript.

How biological inductive bias can be quantitatively described so it can be easy to understand the claim "Architectural Bias influences alignment in behavior?

We note that the exact subtitle ("Architectural Inductive Bias Influences Alignment and Scaling Behavior") was not specifically about behavioral alignment but rather about scaling properties in general, i.e. the “behavior of scaling”. We see how the phrasing can be misleading and will revise it for clarity.

Our results show that biological inductive biases mainly affect the scaling properties of neural alignment, especially in low-data regimes. Specifically, Figure 5b illustrates clearly that behavioral alignment benefits significantly more from task optimization compared to neural alignment. Further, Figure 5a demonstrates that for behavioral alignment, models with strong versus weak inductive biases closely follow the same scaling trajectory, while significant differences remain for neural benchmarks. Indeed, after extensive training, models with weaker inductive biases (e.g., ViTs) even achieve slightly higher absolute behavioral alignment scores than models with stronger inductive biases.

Quantitatively, these biases shift neural alignment scaling more than behavior. Exploring other priors—like local connectivity or predictive coding—is an exciting direction.

Thank you again for your review. We kindly ask the reviewer to consider raising their score and take into account the results that might have been missed in the first pass, such as the diversity of architectures, the novelty of our findings, and the impact on future model building.