Thank you for your review and for highlighting the “sound experimental framework” of our paper. We appreciate the suggestion of further quantifying our results, and have added experiments to address this. See direct responses below.

(1) Re: Replicating results on models trained with multiple random seeds

Yes, another way to validate the differences between the ResNet50 and AlexNet architectures would be to test models trained on different random seeds. The results with the Shape-Image-Net trained network seemed even stronger to us than the results that would be obtained with random seeds, because these models were trained with different datasets and behave differently with natural images. However, for completeness (and also because we had the same question), we ran an experiment with principal distortions for 6 simultaneous models where there are 3 seeds each for ResNet50 and AlexNet. This result is now included as Supp. Fig. 14.

(2) Re: Transformers

We have run an experiment comparing an EfficientNet to a ViT and added this to Supp. Fig. 12. We are also constructing a Jupyter notebook similar to the early visual models, but that generates principal distortions from models in the Huggingface Pytorch Image Models repository (Timm), which we hope would encourage researchers to use our method and explore the rich and diverse set of models available to the community. Please see the general response On the choice of AlexNet vs. ResNet50 for additional details.

(3) Re: Attention maps

This is a very interesting question! It is true that our method and attention map visualization can both be used to interpret model representations. We have added a bit to the introduction and discussion highlighting that our approach is complementary to other techniques for model interpretability (see general response Practical applications and model interpretability for more details). That said, it would be very interesting in the future to run a similar analysis on a large set of transformers with a similar analysis as the foreground vs. background mask comparison (see below) but measure the distortion power weighted by the attention maps. But we leave this for future work.

(4) Re: Qualitative vs. Quantitative Results

We have addressed this by adding a new experiment on images with blurred foregrounds and backgrounds, outlined in the general response. For this experiment, we explicitly chose to generate distortions on a dataset that has multiple images per class (compared to our original distortions, where we only had one image per class) so that we could also look at whether the results were impacted by class. The results are quite consistent regardless of the class, and so we believe this is not a class-specific effect but rather something about the image structure. For more details, see the general response section DNN result quantification.

As another quantification, we looked at how our distortions generated for intermediate representations change the classification decisions of networks (see new Supp. Fig. 13, and also included in Supp. Figs. 12C and 14C. We found that the generated principal distortions drove changes in the final layer output, leading to classification changes with small image perturbations, and the fact that this occurs (even for perturbations generated for intermediate stages) strongly suggests that these distortions are “meaningful” in some way to the classification predictions. In future work, it would be interesting to generate distortions specifically for the classification layers of networks where one might see more differences between distortions generated from images of different classes.

(5) Re: Dataset section

Thanks for the suggestion. We have added dataset sections to the methods describing the image datasets used to generate the principal distortions, one in the early visual model experiment (C.1) and one in the deep neural network experiments (D.1).

Thank you for your careful reading of our paper and for your thoughtful comments, questions and feedback. We have addressed each point below.

W1: Incremental progress over existing metrics

If “metric” is meant in a precise sense (i.e. a notion of distance that is symmetric and obeys triangle inequality), we are unaware of any other practical metric for comparing the local geometry of two image representations. Alternatively, if “metric” is meant in the sense that there are other methods for measuring or comparing the local geometry of image representations, we agree that our work builds on existing methods (Berardino et al. NeurIPS 2017, Zhou et al. Unireps 2023). However, we believe that our extension is critical since it defines a metric (in the precise sense) so that given a collection of models, one can essentially perform a principled dimensionality reduction that allows for comparison of these models in a low-dimensional space (1-dimensional in all of our examples, but our approach can be readily generalized).

W2: Application of FIM to deterministic models

This is a fair point, but we note several reasons why FIM may still be reasonable, beyond the fact that prior work such as Berardino et al. and Zhou et al. already provide precedent for our choice.

First, when comparing deterministic models, we assume isotropic Gaussian noise, so the FIM is the squared Jacobian of the model, a natural object to analyze when computing the local sensitivity of a deterministic model to small input perturbations.

Second, deterministic DNN models have become a prevailing model of the human visual system. It’s long been hypothesized that human perceptual discriminability is based on the ability of downstream brain regions to distinguish between stochastic representations in the visual system. Therefore, it seems necessary to have a unified framework for comparing deterministic models (e.g. DNNs) with human perception (which is based on stochastic representations).

Third, many DNN models are stochastic (e.g. dropout layers, diffusion layers, VAE latents). Although we did not directly investigate these types of representations, our approach would apply naturally to these cases.

W3: Aggregation of metric across stimuli or datasets

Indeed. When we first started, we thought that the ordering of the models, etc would strongly depend on the base image. It’s quite remarkable how consistent the orderings of the models is (e.g., see the error bars in Fig 3E). Therefore, even though each pair of principal distortions is only measuring the local geometry of the model at a specific point, they appear to be consistent across stimuli for the experiments presented in our paper. See more discussion in Q2 below.

Q1: Quantitative results

Please see section DNN result quantification of our general response

Q2: Metric depends on the stimuli

Absolutely. The “distance” between models is at a specific stimulus and there is no requirement that this distance be consistent across stimuli. The similarity we observe in the paper for different images might reflect architectural biases present in the whole image space. We expect there are other cases where models would behave differently for different stimuli, and in these cases additional experiments would need to be done for interpretation of the distortions. We are actively thinking about this, but leave direct suggestions for future work.

Q3: Interpretability methods

Please see the section Practical applications and model interpretability in our general response.

Q4: Early visual models

Please see the general response section Clarification regarding early visual models for a list of changes (additional text and figures) we made to clarify the section on early visual models.

Q5: Inductive biases

We speculate that these differences in principal distortions are related to architectural differences. For example, we generated principal distortions for comparing layers of EfficientNet and a Vision Transformer (Supp. Fig. 12) and found that the principal distortions that the Vision Transformer is most sensitive to have notable grid artifacts that correspond to the size of the input patches to the model. Full experiments ablating architectural components like residual connections or changing the size of convolutional kernels would be interesting in the future.

Additional Feedback:

FIM for deterministic models

Yes, good point! For deterministic smooth models, the metric on stimulus space induced by the FIM is the pullback of the Euclidean metric in representation space. We added Line 269.

Strengthening discussion

We think that the additions to the paper described in the general responses DNN result quantification, Early visual model clarification and Practical applications and model interpretability address this point.

Introducing quantitative analyses

Please see DNN result quantification in the general response.

Thank you for your interest in our work, and in particular for being interested in how we could use the method to investigate human perception. We have included responses to each question below.

Q1 & Q2. Relationship to human sensitivity and effectiveness of the approach

We agree that the early vision model results would be more complete with a full psychophysics evaluation of the distortion sets, however in the current paper we decided to prioritize highlighting how the method can be used in multiple different settings and with large number of models, rather than focus exclusively on models of human perceptual distortions as was done in (Berardino et al. NeurIPS 2017). Given that other reviewers have mostly highlighted the DNN parts of the paper and asked for more analysis/experiments with those, we believe that holding off on a large-scale psychophysics experiment on the early visual models is more appropriate for a separate paper (we would likely opt for a journal article, rather than a conference paper) where we could “do it right” (and more practically, it is unfortunately beyond our abilities to complete in the rebuttal period).

That said, we clarified the early visual model section to make the connection to future experiments more explicit (see the general response section Early Visual Model Clarification).

3. Practical applications of principal distortions

For early visual models, once again see general response Early visual model clarification. We have also added additional discussion about using the method for DNN interpretability, see general response Practical applications and model interpretability.

4. Generality of the approach (what other networks can be compared)

Our method can be applied to any differentiable model. See the general response On the choice of AlexNet & ResNet50 where we demonstrate the generality of our approach by applying it to more modern architectures. As the method investigates the local geometry of the image representation, it is not dependent on having a classification decision and is equally applicable to models trained for segmentation, detection, or even unsupervised methods. For instance, note that the parameters of the early visual models were optimized in Berandino et al. 2017 for predicting human distortion ratings (see more details in Supp. Section C, starting on line 854).

5. Manuscript is over 9 pages

This year, 10-page manuscripts were allowed for both the submission and rebuttal phases.

Thank you for your comments and suggestions. We think that the analyses based on your questions have significantly strengthened the paper. Direct responses are below.

Model Selection

Thank you for the suggestion! We now compare two modern architectures (EfficientNet and Vision Transformers, Supp. Fig. 12). We are also constructing a Jupyter notebook that generates principal distortions from models in the Huggingface Pytorch Image Models repository (Timm), which we hope would encourage researchers to use our method and explore the rich and diverse set of models available to the community. For more details, see the general discussion section On the choice of AlexNet & ResNet50.

Practical Utility and Generalizability

Please see our general discussion section Practical applications and model interpretability for a discussion on the practical utility of our method for neural network interpretability and a list of specific changes to the paper. We have also strengthened the paper with additional empirical results that demonstrate our method’s generality (see general response DNN result quantification). These results include quantification of our previous results on the locations of principal distortions of the ResNet50 and AlexNet architectures (Supp Fig 15), a demonstration that the perturbations affected the network classifications (Supp Fig 13), and (as noted above) an application of our method to modern architectures (general discussion section On the choice of AlexNet & ResNet50).

Thank you again for taking the time to review our paper. We've demonstrated that our method generalizes to modern architectures (see Supp Fig 12) and elaborated on the practical relevance of our approach in the Practical Applications and Model Interpretability section of the general response. Please let us know if we have addressed your concerns.

Thank you for your comments, questions and suggestions. We have updated text and included some additional analyses to clarify the method and strengthen the paper, each detailed below.

W1: Effectiveness of the FIM as a metric.

FIM has a long track record in computational neuroscience (e.g., see the references on Lines 155-156) and more recently in ML (e.g. Berardino et al. NeurIPS 2017, Zhao et al. “The adversarial attack and detection under the Fisher information metric” AAAI 2019). This is because it has several appealing properties such as providing a lower bound on the ability of an ideal observer to differentiate between two stimuli, which is especially relevant when comparing to human perception where discrimination thresholds are hypothesized to reflect the ability of downstream brain areas to distinguish between stochastic representations.

W2: Validation plan to confirm the accuracy or reliability of the findings.

Please see the section DNN result quantification of our general response.

Questions:

How is $f({\bf s})$ computed in practice?

The function $f({\bf s})$ corresponds to any differentiable model; e.g., $f({\bf s})$ could correspond to the activation of a DNN layer in response to an input ${\bf s}$. We have edited the text on lines 168-169 to further clarify this point.

What if ${\bf I}({\bf s})$ is degenerate?

Yes, the metric is ill-defined if one of the FIMs is degenerate. A straightforward solution is to regularize the FIMs by adding the identity matrix scaled by a small constant. We clarify this point on lines 961-964.

How are two image representations obtained from a single image ${\bf s}$? Are ${\bf I}_A({\bf s})$ and ${\bf I}_B({\bf s})$ learned by different neural networks?

In our experiments, each image representation corresponds to a model's response to a single image. For example, in the DNN experiments, a "model" corresponds to the activations in a layer of a DNN. For a full list of the "models" we used, see Sec. C (early visual models) and Sec. D (DNN layers) of the supplement. Note that we use pre-trained networks and the model weights remain fixed. We now emphasize this on Line 352.

The FIMs ${\bf I}_A({\bf s})$ and ${\bf I}_B({\bf s})$ are computed, not learned. For example, for deterministic models $f_A({\bf s})$ and $f_B({\bf s})$, we assume isotropic Gaussian noise in which case the FIMs are ${\bf I}_A({\bf s})={\bf J}_A({\bf s})^\top{\bf J}_A({\bf s})$ and ${\bf I}_B({\bf s})={\bf J}_B({\bf s})^\top{\bf J}_B({\bf s})$, where ${\bf J}_A({\bf s})$ is the Jacobian of $f_A(\cdot)$ at ${\bf s}.$

How are the coefficients for $\boldsymbol{\epsilon}_1$ and $\boldsymbol{\epsilon}_2$ determined in the proposed approach?

We assume you are referring to Figure 2 (if not, please let us know). Please see the general response Early visual model clarification and updated lines 315-322 of the text.

The interpretation and justification of the estimated principal distortions are difficult to justify.

We now include quantification of our qualitative observations (see our general response DNN result quantification). Based on your suggestion, we tested how the addition of principal distortions to the base image changes the classification decision of each network (Supp. Fig. 13). We find that each generated principal distortion drove changes in the final layer output of one DNN (leading to classification changes with small image perturbations) but not the other DNN. This analysis raises very interesting future directions using the principal distortion method directly applied to classification outputs of networks, and potentially creating a form of adversarial example that maximally differentiates many classification outputs. Detailed analysis of principal distortions applied to classification layers is beyond the scope of the current paper which introduces the method, but would be quite interesting as a future direction. We appreciate your suggestion and think it opens up very interesting directions for future exploration!

Difficulty connecting the identified distortions with the concept of local geometry.

The local geometry at a base image is closely related to how sensitive the model responses are to small distortions of the base image (see Sec. 2.1). Therefore, a model’s sensitivities to principal distortions capture aspects of the local geometry induced by the model. Once these principal distortions are computed, we focus on identifying properties that consistently characterize them. Ultimately, by characterizing the principal distortions, we hope to better understand the differences in local geometries of models (e.g. DNNs). Please let us know if this helps clarify our proposed approach. We have also added a line to the introduction Lines 46-47 to help readers link local geometry to distortions.

Thank you once again for taking the time to review our paper. We've elaborated on our choice of the Fisher information matrix as a metric, as well as quantified the reliability of our findings. Please let us know if we have addressed your concerns.