We are grateful for the reviewer's constructive and insightful review of our work.

1. Interpretation of Results in High-Data Regime. We agree that in the high-data regime, misidentification affected smaller convolutional models, which tend to have lower capacity and were trained on smaller datasets, potentially yielding less distinctive representations. However, in a follow-up analysis including ten additional models (see our response to Reviewer Suzu), we found that even OpenCLIP-ConvNeXt-XL was often misidentified. This finding suggests that misidentification in the high-data regime may not be limited to older or lower-capacity models. Importantly, model confusability is context dependent: it is a property of model pairs rather than of individual models.

This also raises a second concern. Suppose the read-out human representation is more similar to that of a simpler or smaller model. If flexible evaluation procedures tend to favor higher-capacity models because they can better approximate arbitrary targets, these procedures may systematically mislead us. In such cases, richer models could outscore simpler but more human-like ones—not because they are better aligned in their native geometry, but because they are more easily aligned post hoc. Therefore, we do not find the fact that it is mostly smaller CNNs being misidentified to be entirely reassuring.

Due to space constraints in the current draft (even with the additional page), we cannot commit to a specific revision of the discussion section at this stage. However, we will incorporate this more nuanced interpretation, including the reviewer’s observation, into the revised version.

2. Measure of Effective Dimensionality. We thank the reviewer for the helpful suggestion to explore non-linear dimensionality estimates, which we were glad to incorporate.

Following this suggestion, we estimated intrinsic dimensionality (ID) using Generalized Ratios Intrinsic Dimensionality Estimator (GRIDE) (Denti et al., 2022) through the publicly available DADApy implementation (with GRIDE computed across neighborhood sizes k = 1–64).

We computed ID for each model’s representation before and after alignment, and reran the regression analyses described in Table S3 in the manuscript, replacing PCA-based effective dimensionality with the GRIDE measure.

Below are the updated results:

(ID: GRIDE intrinsic dimensionality estimate)

Intrinsic dimensionality—whether measured on original or transformed features—was not a significant predictor of the model recovery outcome (p > 0.15), whereas the alignment-induced representational shift remains the dominant, statistically significant predictor. Note that our significance threshold accounts for multiple comparisons, making the test relatively stringent.

We will include the updated GRIDE-based results as a supplemental figure in the revised manuscript.

3. Broader impact. To avoid redundancy, we kindly refer the reviewer to our response to the second point of reviewer Suzu. In brief, we believe that while identifying models that mechanistically explain human brain and behavior is a central ambition in neuroAI and cognitive computational neuroscience, current benchmarking practices tend to reward predictive performance alone—often without asking whether the evaluation metrics can reliably identify the correct model. Our contributions are threefold: (1) we characterize the severity of this issue in an important benchmark; (2) we introduce a general approach for evaluating model recovery accuracy using noise-calibrated simulations; and (3) we aim to raise awareness of the tradeoff between predictivity and identifiability, advocating for model recovery simulation as a standard component of the analytic pipeline when comparing computational models.

4. Misidentification due to stochastic optimization and $W$ matrix initialization. In all experiments, we optimized the matrix $W$ using full-batch L-BFGS and initialized $W$ to the identity matrix for both the data-generating and candidate models. Given fixed data, initialization, and optimizer hyper-parameters, full-batch L-BFGS is deterministic; consequently, the optimizer does not introduce stochasticity. The only stochastic elements in our pipeline are the triplet-sampling procedure described in Appendix A.3, and the sampling of the simulated responses from the discrete probability distribution defined by the data-generating model. The random seed was varied across repeats of the same experimental condition.

6. Human test-retest reliability. We agree with the reviewer that studying inter- and intra-subject response variability would be an insightful direction for future work, and we are actively looking into this question. The current noise ceiling estimate reflects both sources of variability.

We believe the reviewer would agree that model recovery accuracy and inter- and intra-subject response reliability are related yet operate on different scales: even if the behavioral responses are unreliable (e.g., 50%), given many such responses, model recovery accuracy can still approach 100%. For example, consider comparing two biased coins, one with $p=0.5$ and the other with $p=0.55$. A single coin toss is not very informative, but many independent tosses can reliably reveal which of the two coins we are holding.

We thank the reviewer for their thoughtful response and deep engagement with the manuscript.

Generalization to neural data. We agree that generalization to neural data is non-trivial. Model identification using neural data operates in a markedly different regime: responses are multivariate and continuous rather than univariate and discrete.

Following the comments of SokD and other reviewers, we carefully considered the possibility of conducting a simulation of model recovery using neural data with a realistic neural noise model. We concluded that this would be a substantial undertaking. A realistic simulation would entail accurately capturing both single-trial variability and inter-channel noise correlations. For the NSD fMRI dataset, a complete reanalysis would be required because GLMSingle does not provide statistically independent single-trial response estimates (across-trial reliability is used to guide HRF choice and signal denoising). For MUA or spike-rate data, access to single-trial responses is easier, but modeling must account for the multivariate, non-Gaussian distribution of the noise (i.e., faithfully capturing response variability and noise correlations). Furthermore, a realistic simulation may require modeling modality-specific noise sources, such as head motion in fMRI, or slow drift in response amplitudes in electrophysiology.

Hence, we believe that such a study would justify a separate paper devoted to this problem. Combining Things-Odd-One-Out and one or more neural datasets into a single paper would considerably broaden the scope. However, it would not allow for sufficient detail and depth in describing the various analytic choices required for each simulation study. Here, we chose to focus on a single but important test case and to pursue it in as much depth as possible. We hope we can gain the reviewer's support for publishing the present work in NeurIPS, despite our deliberate focus on a behavioral test case at this stage.

We have revised our draft to reflect the uncertainty about the generalization to neural data. Revised limitations paragraph:

The scope of our simulations is limited to behavioral data, and specifically, to the THINGS odd-one-out task. We chose this task as a test case because it is supported by a large empirical dataset (Hebart et al., 2023) and allows straightforward simulation of synthetic responses by sampling from model-specified multinomial distributions. The noise-calibrated simulation approach can be readily extended to other behavioral paradigms, such as classification (Battleday et al., 2020) or multi-arrangement (Kriegeskorte & Mur, 2012). Model identification using neural data operates in a markedly different regime: responses are multivariate and continuous, rather than univariate and discrete as in the behavioral case. Therefore, while our results demonstrate a pronounced predictivity–identifiability trade-off when comparing models to behavior in a large dataset, the severity of this trade-off for neural data cannot be inferred from our findings.

We also added the word "behavioral" to the abstract's conclusion:

These findings demonstrate that, even with massive behavioral data, overly flexible alignment metrics may fail to guide us toward artificial representations that are genuinely more human-aligned.

We wish to keep the sentence "flexible evaluation may incur a hidden cost". It presents our working hypothesis rather than the paper's empirical conclusion.

Link to recent neuroscientific reports on flexible evaluation. We agree that these links are important, and in part, they have motivated our work. The remainder of our new limitations paragraph spells this out.

Recent reports of qualitatively distinct neural network models achieving indistinguishable performance under flexible comparisons to neural data (Conwell et al., 2024; Storrs et al., 2021) make this question especially pertinent. Addressing it will require future work using noise-calibrated, modality-specific neural simulations.

Note that the equivalent model performance reported in Storrs et al. and Conwell et al. does not prove that the compared models make equivalent predictions. Similar accuracy is a necessary but insufficient condition for prediction equivalence. For example, models with similar aggregate performance may produce different error patterns (see Rajalingham et al., 2018, J. Neurosci.) or explain different subsets of the data. Therefore, comparing each model to neural or behavioral data does not eliminate the need to study how distinguishable the predictions of different models are, as we do here.

Additional criteria for evaluation measures.

This is a good point. Given the limited space, we incorporated it by expanding the second future direction in the discussion:

Constraining data-driven model alignment by biologically motivated and/or inter-individual variability–informed priors (...) may improve upon the overly flexible family of linear transformations. Furthermore, imposing greater constraints on the readout may enhance its interpretability. For example, constraining the learned stimulus embeddings to be non-negative prevents features from canceling each other (e.g., Mahner et al., 2025). Finally, Bayesian readout models, which estimate a distribution of feature weights rather than a point estimate, may improve robustness to sampling noise.

Nonlinear and structured linear probes. We would be happy to introduce additional probes into our analysis. We will update the next version of the manuscript regardless of whether it is a camera-ready version or a resubmission. We have already introduced a more traditional Frobenius-norm penalty on the weights and found that it is slightly less effective for model identification than the penalty introduced in the paper.

However, we would be grateful for clarification on the suggested probes. Could the reviewer clarify what kind of nonlinear probes they envision for this task? The vast majority of the literature purposely relies on linear probes to ensure interpretability. Regarding structured linear probes, since we are modeling higher-order perceptual judgments, penultimate layers are used (following Muttenthaler et al., ICLR 2023). This rules out separable readout (i.e., distinct weights for feature channels and spatial locations). The remaining probes seem to be sparse. Does the reviewer have in mind probes that are sparse in the linear transformation matrix, or probes that are sparse in the resulting embedding matrix (as in the SPoSE and VICE embedding models)? If it is the former, are there any examples in the literature where sparse readout is motivated and implemented?

Distinguishing Predictive Performance from Model Identification. All of the human–DNN comparison studies cited in the manuscript are aimed at understanding the brain, rather than predicting responses as an end in itself. However, we agree that—especially in the context of NeurIPS—it is worthwhile to explicitly distinguish between prediction for its own sake and prediction used as a means for model identification. We have edited the introduction to make this distinction and to cite several studies that use flexible evaluation of behavioral model predictions, with the aim of understanding the underlying biological computations:

When evaluation is made flexible by fitting linear weights to improve the alignment of neural network representations with brain [...] or behavioral data (e.g., Peterson et al., 2018, Cognitive Science; Battleday et al., 2020, Nature Communications; Daube et al., 2021, Patterns; Muttenthaler et al., 2022, ICLR), this approach achieves predictive accuracy that exceeds that of any other computational model. In some neuroscientific applications (e.g., brain--computer interfaces), accurate prediction is useful regardless of the underlying mechanism. In contrast, basic-science studies in computational neuroscience often rely on the assumption that a neural network whose representations are more predictive of brain or behavioral data is a better model of the mechanisms underlying the biological data.

We thank the reviewer for their thoughtful and supportive evaluation of our work. Below we address each point in turn.

1. Limitation to behavioral data We appreciate the reviewer's endorsement of behavioral data as an appropriate scope for a first noise-calibrated simulation study of this problem, and we intend to extend this work to neural data in follow-up research. We refer the reviewer to our reply to Reviewer SoKD for more details on how we understand the relationship between the behavioral and neural domains in this context.

2. Noise ceiling estimation details. We thank the reviewer for the suggestion to make this description clearer. To estimate the noise ceiling, the THINGS odd-one-out dataset includes a subset of 1,000 triplets—each presented to approximately 30 participants. To compute a lower-bound on the performance of the ideal model, we used a leave-one-subject-out procedure on this data, following Hebart et al., 2020 (Nature Human Behaviour). For each triplet and each participant, we withheld that participant’s response, determined the majority choice among the remaining responses, and recorded whether the held-out response matched the majority. We repeated this for all participants, averaged the match rates across triplets. This reproduced the noise-ceiling values reported by Hebart et al. We have added the following paragraph under “Calibration” in the Methods section:

We estimated the noise ceiling with a leave-one-subject-out procedure: for every triplet, we removed one participant’s response, took the majority vote of the remaining responses, and noted whether the held-out answer matched that vote. We repeated this step for all participants and averaged the match rates across all triplets.

We thank the reviewer for their positive evaluation and detailed review.

In response to: “The study is restricted to one behavioral task of odd-one-out judgments and one dataset. The quantitative results are specific to this particular model set and may not generalize broadly.”

We selected the THINGS odd-one-out paradigm because its massive scale (4.7 million human judgments) provides favorable conditions for identifiability, and its simple three-way response paradigm allows precise probabilistic simulation. Therefore, we believe that model recovery accuracy plateauing below 80% even with millions of simulated trials indicates a fundamental limitation of flexible linear evaluation, rather than a quirk of this dataset.

We agree that the particular composition of the model set may influence recovery outcomes. To test this possibility, we conducted additional simulations after the submission deadline, using an extended model set. This set included the 20 models reported in the manuscript along with OpenCLIP ConvNext XL, ALIGN, FLAVA Full, SigLIP2 B, BLIP 2, EVA02 CLIP-enormous, PE Huge, ImageBind Huge, OpenAI CLIP ViT Large, and AlexNet. When curating the new models, we prioritized multi-modal (visual-language) models, motivated by the strong performance of CLIP in the original evaluation.

Below, we report the model recovery performance of the extended model set as a function of dataset size:

Notably, OpenAI CLIP ConvNeXT XL was not consistently identified when it was the data-generating model. We will incorporate these simulations along with a detailed confusion matrix in the revised (camera-ready or resubmitted) version of the paper. Overall, the results indicate that model confusability is not limited to the original 20-model set reported in the current version of the manuscript.

In response to: The core insight that flexible metrics can overfit isn't entirely novel; this has been recognized in various forms across ML. The specific quantification in this behavioral context is useful but somewhat incremental. I wonder if this work would be better suited for a more specialized venue focused on computational cognitive science or model evaluation methodologies rather than NeurIPS.

We agree with the reviewer that the insight that flexible evaluation can lead to overfitting is well established in statistics and machine learning. It is also increasingly acknowledged in neuroAI and cognitive computational neuroscience. However, most neuroscientific studies comparing DNN and neural/behavioral responses place strong emphasis on maximizing predictivity, and seldom evaluate how well their chosen alignment metrics support model identification. Here, we analyze model identification performance in a leading benchmark—the largest currently available behavioral dataset—using a realistic simulation and a continuum of relevant evaluation metrics. To our knowledge, no prior work has systematically quantified model identification performance under realistic, noise-calibrated conditions. While our analysis focuses on a specific benchmark, the methodology we introduce and the substantial identifiability–predictivity trade-off it uncovers are relevant to a broader class of model-evaluation problems. We hope that this work will encourage researchers to conduct such analyses as an integral part of their empirical studies.

We agree that this manuscript could also be suitable for specialized venues such as CCN or NBDT. However, many researchers active in neural network modeling of brain and behavior are more likely to be exposed to a NeurIPS paper. Several studies that rely on flexible metrics for model identification have themselves been published at NeurIPS/ICLR/ICML. Publishing this work at NeurIPS can help ensure that the identifiability–predictivity trade-off is recognized and addressed by the broader modeling community.

In response to: “Your study only looks at odd-one-out similarity judgments, which is pretty narrow. Do you think these problems with model recovery would show up in other kinds of behavioral tasks or even neural data?"

As discussed in our response to the reviewer's first point, we do not see a reason that other behavioral tasks would yield better model identification performance. Moreover, existing real behavioral datasets are at least an order of magnitude smaller than THINGS-odd-one-out, and therefore even more prone to model misidentification.

As for neural datasets, the statistical regime is different—the responses are multivariate and continuous and rather than univariate and discrete. We are therefore more cautious about making strong predictions about these settings without additional data. Please see our response to Reviewer SoKD below for a more detailed discussion of this point, as well as a relevant modification of the manuscript.

We thank the reviewer for their interest in our manuscript and for the constructive review.

1. Clarifying the model-recovery simulation using a student–teacher analogy. We appreciate the suggestion and have incorporated this analogy into our Methods section:

This setup is analogous to data distillation: the candidate models act as "students" attempting to approximate the input–output function of a "teacher"—the data-generating model. Model recovery is successful when the best-performing student is the teacher itself, rather than an alternative model.

2. Elaborating on proposed avenues to address the issue identified in the paper. We outlined several possible research directions at the end of the Discussion section, under the heading "Navigating the accuracy–identifiability tradeoff." In response to this and related comments, we have revised and expanded the section, further elaborating on possible approaches to mitigate the issue identified in our work.

1. Change the stimuli: As in many model comparison studies, we evaluated models out-of-sample, that is, on new stimuli drawn from the training distribution. Out-of-distribution generalization, which more strongly probes the models' inductive biases, may offer greater model comparative power. Stimuli designed to elicit model disagreement may yield even greater gains. The recovery gains we obtained from larger and more diagnostic triplet sets suggest that smarter sampling matters at least as much as sheer volume (Fig. 1). Adaptive, model-driven stimulus selection constructing trials that maximize the expected divergence in network responses—can sharpen our ability to treat predictive accuracy as an indicator for human—model alignment, enhancing current flexible alignment methods without compromising identifiability.

2. Change the metrics: Constraining data-driven model alignment by biologically motivated and/or inter-individual variability–informed priors may improve upon the overly flexible family of linear transformations. Furthermore, imposing greater constraints on the readout may enhance its interpretability. For example, constraining the learned stimulus embeddings to be non-negative prevents features from canceling each other (e.g., Mahner et al., 2025). Finally, Bayesian readout models, which estimate a distribution of feature weights rather than a point estimate, may improve robustness to sampling noise.

3. Change the models: The considerable geometric shifts required to align the networks suggest that linear probes can obscure important representational mismatches. Embedding richer priors directly into the models—through task design, objective functions, or biologically inspired architectures—could allow aligned representations to emerge natively, reducing the need for substantial post hoc transformations. More broadly, further progress in neural network-based modeling of brain and behavior may depend less on ever-larger data-driven fits of pre-trained models and more on deliberate model refinement to embody explicit computational hypotheses.

3. The effect of training objectives on model recovery accuracy. We thank the reviewer for highlighting this for our consideration. After the submission deadline, we introduced ten additional models, primarily image–text-aligned (e.g., ALIGN, OpenCLIP-ConvNeXt-L, OpenAI-CLIP ViT-L) and reran the model-recovery simulations. To address this specific comment, we estimated a between-objective confusion matrix, in which model recovery is considered incorrect only if it confuses models trained on different objectives. However, even under this permissive criterion, model recovery plateaued at 72%, compared to 70% under the standard criterion applied to the same model set. We have added a summary of this result to the Results section of the manuscript:

Model-recovery performance plateaus despite objective-driven representational divergence Recent work (Lampinen et al., 2024) has shown that different training objectives produce distinct representational biases. We categorized each model as supervised, unsupervised, or image–text-aligned and then performed a between-objective model-recovery analysis. Even with a 4.2-million-triplet training set, objective-based grouping yielded only a modest accuracy gain: from approximately 70% to 72%. These results indicate that, although initial internal representations reflect objective-specific biases (Lampinen et al., 2024), flexible evaluation largely obscures these differences.

4.Extension beyond behavioral settings. We expect that the predictivity–identifiability trade-off will likewise manifest in neural data. However, the severity of this trade-off may change, because model comparison using neural data operates in a somewhat different regime: responses are multivariate and continuous, rather than univariate and discrete. We have revised the discussion section to reflect this uncertainty:

The scope of our simulations is limited to behavioral data, and specifically, to the THINGS odd-one-out task. We chose this task as a test case because it is supported by a large empirical dataset (Hebart et al., 2023) and allows straightforward simulation of synthetic responses by sampling from model-specified multinomial distributions. The noise-calibrated simulation approach can be readily extended to other behavioral paradigms, such as classification (Battleday et al., 2020) or multi-arrangement (Kriegeskorte & Mur, 2012). Model identification using neural data operates in a markedly different regime: responses are multivariate and continuous, rather than univariate and discrete as in the behavioral case. Therefore, while our results demonstrate a pronounced predictivity–identifiability trade-off when comparing models to behavior in a large dataset, the severity of this trade-off for neural data remains an open question.

We aim to resolve this uncertainty in follow-up work. To reduce the risk of misinterpreting the scope of our empirical results, we added the word “behavioral” to the concluding sentence of the abstract:

These findings demonstrate that, even with massive behavioral data, overly flexible alignment metrics may fail to guide us toward artificial representations that are genuinely more human-aligned.