Representational Similarity via Interpretable Visual Concepts

[73oU-3.3] Computational cost trade-off.

For a given class and layer, comparing two models using MCS only requires using Pearson correlation whereas using RSVC requires training five lasso regression models. We give more details about computational cost in Table A.3 and Section B.

[73oU-3.4] Pearson and Spearman

While we compute plots for both Pearson and Spearman correlation, we find little difference with the two methods. Therefore, all plots use Pearson correlation. We will include a comparison in the appendix in the final version.

[73oU-3.5] Clarification of motivation for maximum in MMCS.

Your understanding of the reason for the maximum is correct. We add some text clarifying this point in the updated version of the manuscript (Line 214-216).

[73oU-3.6] How is lambda selected?

We have conducted an ablation of the L1 penalty in Fig. A12 of the appendix. We find that the histogram of similarity values is consistent for lamba values 0.01, 0.05, and 0.1, but that similarity values drop significantly when going to 0.5. In our work, we choose 0.1.

[73oU-3.7] Why NABirds for DINO and MAE.

We wanted to explore a second dataset (NABirds) that had less images available for training the regression model. We also include several ablation experiments (Secs. A.3, A.4; Figs. A4, A5, A7) on the NABirds which are feasible due to the smaller size of the dataset.

[73oU-3.8] How are correlation plots generated?

The correlation plots are for a single, specific concept (a column in $U_1$). We indicate this on the x and y labels with a vector $u^a_1$ and $u^a_{2\rightarrow 1}$ where $a$ represents an arbitrary column vector from $U$ and it is also noted in Line 079.

[73oU-3.9] Differences in Fig. 2 are not clear.

Some reviewers found Fig. 2 to be challenging to interpret. We swap it with a sample from the barbell class that has much clearer differences. There are more examples provided in the appendix. We also conducted a new experiment using ChatGPT (Sec. A5, Fig. A6) to provide interpretations given two image collages. We ask it to provide an overview of both similarities and differences and find that it roughly matches the manual annotations we provide.

[73oU-3.10] Defining/grouping low, medium and high importance.

in Section A.3 and Figs. A4, A7, we have added an experiment exploring the impact of training with and without extra data on the features learned by the network. In these experiments, we have detailed a more systematic approach to surfacing interesting low similarity features.

[73oU-3.11] Number of points in Fig. 3

There are 10 concepts per class and 1000 classes for the ImageNet experiments, resulting in 10000 points. There are 555 classes and 10 concepts per class for the NABirds experiments, resulting in 5550 points. We have also added color bars indicating the number of points vs. color intensity.

[73oU-3.12] Interpretation of Fig. 3 (medium importance vs zero/low importance).

The x-axis represents similarity and the y-axis represents importance. As you have noticed, density along the x-axis peaks around medium similarity. On the y-axis, while there are many zero/low importance concepts it is clear in Fig. 4 (replacement test) that these concepts do not impact model behavior. Therefore, we believe it is fair to see that medium importance concepts drive the bulk of model behavior.

[73oU-3.13] Separating Fig. 4 into subplots.

We have not yet created this plot, but will include it in the final version.

Thank you for your review.

[73oU-1] Unclear motivation of work. Expected a more systematic exploration of changes in features as scale, data and architecture is varied.

In this revision, we clarify the motivation of our work in the abstract, introduction, and related works. Our primary contributions are (1) defining the problem of visualizing model differences, (2) introducing a new method, RSVC, to address the problem, and (3) establishing that RSVC measures what we intend it to measure. We have updated the manuscript to clarify the importance of this problem for the field of XAI. We believe that by introducing this problem and our approach, the XAI community should have an exciting, new way to think about what makes a good explanation for models. We expect that future research will improve the method and apply it for deeper insights about model differences. We also make some initial steps towards applying it on more systematic variations during training and exploring some of the features that are discovered. In these new experiments (Sec. A.2, A.3, Figs. A.2-A.4 A.7) , we manipulate the training of a ResNet18 on NABirds and show how similarity changes. We find that increasing differences in training protocol results in progressively more dissimilar and important concepts. We explore one pair of models in these experiments and generate qualitative visualizations of feature differences. We detail these experiments in the response to reviewer [Wky2-1] .

[73oU-2] Why conduct the experiments in Figs 3, 4, and 5?

One of the primary goals of this work is to establish the validity of the comparison method. We include these figures because they answer critical questions that assess the validity and usefulness of the proposed tool. Fig 3 shows that RSVC can discover concepts that are unique and important to a model, Fig 4 links representational differences in two model’s to specific changes in a model’s behavior, and Fig 5 (1) verifies existing observations in the literature (2) provides a different perspective into how layers relate in terms of concepts. As a small step in trying to gain deeper insights, we provide a new experiment in the Appendix A3 comparing a model trained on NABirds to a model trained on NABirds and Stanford Cars. We find that there are some consistent differences in the types of features learned. The model trained on both NABirds and Stanford Cars exhibits complex concepts that are semantically aligned with specific car models. For example, one discovered concept is sensitive to the racing stripe on a type of muscle car. In contrast, car concepts for the model trained solely on NABirds appear to be sensitive to a particular color and style of car, but are not specific to the car model. RSCV opens the door to future focused evaluation on a specific model and a small set of classes to present a complete picture of model behavior. In this work we have focused on a broader exploration of the tool's properties to demonstrate its potential.

[73oU-2.1] What does Fig 5 demonstrate?

Fig 5 provides value in a few ways: (1) it shows a subtle, but interesting phenomena suggesting that concept similarity is lower in middle layers than in the final layers, (2) acts as an important sanity check on our method and (3) provides insights into layerwise similarity when comparing DINO and MAE pre-trained models. Points (1) and (3) are new results that were not explored in Raghu et. al. 2021.

[73oU-2.1] Is the L2 replacement test in Fig 4 obvious?

The replacement test helps assess whether prediction error for a regression model is meaningful, since error could be localized to images in which the concept contributes very little to model activations. While the result for L2 is expected, the main point of including this figure is to look at how it changes the logits (kl-div) and match accuracy. In the figure, we include L2 for completeness.

[73oU-3.1] Abstract states it is a practical tool for model failures, but not discussed in the paper.

We have modified the language in the abstract and introduction. Our method provides hypotheses for differences in model behavior, which paves the way for future work to provide a complete explanation for model failures.

[73oU-3.2] Implementation details.

We expand on implementation details in the Appendix (Sec. B). Images are resized to 224x224 and patched into 64x64 patches (evenly spaced), resulting in 16 patches per image. Each patch is resized back to the original image resolutions. Models are always trained with either inception style cropping or random resized cropping, ensuring that patches are in-domain for the network.

Thank you for the review.

[fUV9-1] Missing implementation details.

We provide further details about the approach in the revised Appendix B. We respond to specific questions below.

[fUV9-2] Interpretation of BrainScore.

We see how our text may be considered misleading. Our aim is to highlight the uniqueness of the replacement test, which allows us to systematically test the impact of replacing columns of M2's coefficient matrix, with predicted coefficients from a regression model trained on M1's activation matrix. This test allows us to understand how conceptual differences in model representations lead to functional changes in model behavior. This kind of systematic variation is not conducted as one of the metrics in the BrainScore. We have modified the text in the introduction (first paragraph) to avoid making an incorrect claim about prior work. Yes, brain score compares ANNs to BioNNs, and we used the term NN as a general term to catch all types of neural network based computations. We add text to clarify this point (line 272).

[fUV9-3] Add colorbars to Figure 3 and 4.

We add colorbars to both figures.

[fUV9-4] Mention why L1 is used.

We clarify the motivation for L1 regularization in equation 5.

[fUV9-5] Introduction is brief, please expand.

We have expanded on the introduction to emphasize the motivation for our work, i.e. the potential impact of our method on explainable AI. Also, a discussion on local (saliency maps) and global (concepts) is provided in Section 2.2

[fUV9-6] Better explain the image sets and patch size details.

The number of images used to construct $I^c$ varies depending on the class and dataset. We enforce that there must be at least five images and at most we sample 200 images for concept comparisons. We use a patch size of 64x64 pixels resulting in 16 patches per image. Overlap is not required, since we take the union over the prediction sets from each model. You are correct that number of images, image grouping, and patch size would impact the concepts that are discovered. Future work should scrutinize the selection of concept proposal images further. However, in this work we use an established protocol for concept generation (CRAFT) since our focus is on developing a tool for comparing concepts. We include these details in Section B.1 of the Appendix.

[fUV9-7] What are the train and test splits for regression?

For training the regression model, we use the train dataset of the deep neural network. Images in $I^c$ are split into five folds and five regression models are trained on each combination of four folds. For the final evaluation, the weights of all five regression models are averaged and the regression model is tested on images from the test set of the deep neural network. These details are available in Section B.2.

[fUV9-8] Intuition on grouping when models are not trained on the same set of classes.

In this work, we choose to group images according to the model's predicted class. This choice is made because we are interested in features that contribute when the model believes an image belongs to a particular class. However, there are other reasonable options. For example, images could be grouped using unsupervised clustering in the latent space of the model.

[fUV9-9] Exploration of regression weights.

This is an interesting question, we have not yet explored this in this version of the manuscript. We will provide an update if we are able to complete this exploration. In addition, other reviewers found the volleyball class in Fig 2 to have a complex interpretation, so we have swapped it with a simpler example from the appendix that demonstrates a concept difference on the barbell class. We also have added several new experiments and generated concept difference visualizations for these experiments. Please see [Wky2-1] for details.

Thank you for your review.

[RkXK-1] Sensitivity to concept extraction method.

We agree that the approach may be sensitive to the concept extraction method. In this work, we used a concept extraction approach that had been previously tested in CRAFT (Fel et al. CVPR 2023) and focused on developing a tool to compare the concepts. In Fel et al. NeurIPS 2023 , the authors show that NMF has desirable trade-offs for several properties: reconstruction error, sparsity, stability, FID, and OOD generalization. We have conducted an ablation of the L1 penalty in the appendix. We find that the histogram of similarity values is consistent for lamba values 0.01, 0.05, and 0.1, but that similarity values drop significantly when going to 0.5. In our work, we choose 0.1.

[RkXK-2] Computational cost.

We include a breakdown in Table A3 in the Appendix (Sec. B) detailing the computational cost of each step of our approach. Overall, comparing a ResNet-18 to a ResNet-50 using both RSVC and MMCS takes around 1.2 minutes per class and 20 hours for all 1000 classes of ImageNet on a single machine with a 4090 GPU. We also include details about the number of images used in each step of the approach in Section B.

[RkXK-3] Usability and automation of concept difference interpretation.

We include a new experiment in Appendix A5 using ChatGPT-4o to interpret the image collages generated by our approach. We find that the interpretations provided by the LLVM are similar to the manual annotations we have provided previously.

[RkXK-4] Missing related work and typo.

Rosetta Neurons (Dravid et. al 2023) are most similar in spirit to our work. This method correlates the activation patterns of neurons in two different networks. They seek out neurons with similar activation patterns and coin these neurons to be “Rosetta Neurons”. However, they do not aim to provide a method to test for dissimilar neurons that significantly contribute to model behavior, which is the primary focus of our work. In addition, Rosetta Neurons are found through correlation, which can miss many-to-one mappings from one model to the other. Network-to-Network translation (Rombach et. al 2020) uses an invertible neural network to map between the latent space of a language model and a generative vision model. Their approach focuses on building a tool for text-to-image generation. Our work targets a different problem (explainability) and is designed to accurately quantify and visualize the difference between two models. Thank you for pointing out these references, we include them in the updated manuscript and discuss the differences between our method and Rosetta Neurons (Line 111). We have corrected the typo as well.

[RkXK-5] Why does the approach use per-class concepts?

We use per-class concepts because it helps organize and simplify concepts for the user, however, other grouping strategies are valid and interesting directions. As you have pointed out, grouping across the dataset can help reveal shared computations. In the future, we would like to explore grouping commonly confused classes to discover more complex model behavior.

[RkXK-6] The current approach assumes the models are classifiers.

For non-classification problems users would have to select a method to group images. A simple option in a setting where there are no class labels is to perform unsupervised clustering in the latent space to find clusters of semantically related images. Alternatively, one can use a probe classification dataset to compare SSL representations, as we do for the DINO and MAE models in Figures 3 and 5.

[RkXK-7] What data is used to compute the linear mapping in line 282?

Images for the linear regression are sampled from $I^c$. We provide further implementation details in Section B.2.

Thank you for the review.

[Wky2-1] “Lack of sanity checks and proof of concepts”.

We agree that establishing the validity of the approach is very important. Following the reviewers’ suggestion, we conduct six new experiments that we detail below. These experiments can be found in the appendix and are marked in blue.

(1) Toy Concept Experiment (Sec. A.2, Figs. A.2, A.3) : We train two models from scratch on modified versions of the NABirds dataset. Model 1 (M1) is trained on a dataset in which a pink square is randomly placed on images of the Common Eider class with a 70% frequency. Model 2 (M2) is trained with a pink square appearing 50% of the time on images of any class, giving the pink square no predictive ability. Given this training formulation, we expect M1 to react to pink squares and M2 to ignore them. We apply RSVC to these models and show that (1) we discover a pink square concept in M1 that has a ~0 similarity score to M2 and (2) that other concepts for the Common Eider class are preserved and have higher similarity.

Seed & Dataset Variation (Secs. A.3, A.4, Figs. A4, A5, A7) - We train a ResNet18 model from scratch on the NABirds dataset. The features of this model are used as a baseline for comparisons as other aspects of training are varied. The experiments are ordered by increasing concept dissimilarity.

(2) (Fig. A4) We train and compare to another ResNet18 using a different seed (same initialization). We find that the similarity between the two models is still quite high.

(3) (Fig. A4) We train a ResNet18 on a modified version of NABirds in which the waterbirds (169/555 classes) are excluded. We find that more dissimilar concepts arise, but the overall similarity is still high.

(4) (Fig. A4) We train a ResNet18 on a combined dataset of NABirds and Stanford Cars. We find that the introduction of qualitatively different data, results in learning significantly more dissimilar concepts. We conduct a visual analysis of the features learned from these models. (Fig. A7). We find intriguing differences in the types of features learned. The model trained on both datasets contains complex car features that are specific to a car model. For example, we find a feature that activates for the racing stripe of a specific muscle car. In contrast, we find that the other model uses simpler features that activate for specific colors and car styles.

(5) (Fig. A4) We compare to a model pre-trained on ImageNet and fine-tuned on NABirds. We find that, expectedly, ImageNet introduces a significant amount of dissimilar concepts.

(6) (Fig. A5) We also explore the effects of varying seed when finetuning DINO and MAE models on NABirds. We find that pre-training method has a larger impact on similarity than seed.

We found these results to be intuitive, with conceptual similarity decreasing with larger modifications of the training protocol.

[Wky2-2.1] Do patches make sense? Why not patch the activations after passing the image through the network?

All models are trained with random resized crop or inception style cropping with the smallest crop being smaller than the patch size. This makes the patches in-domain for the network. We clarify this point in the text. It is an interesting idea to patch the activations after passing an image through the network. As far as I am aware, this approach has not been tried. In this work, we made use of an established protocol for concept extraction (CRAFT) that had been tested with human experiments, but other approaches could be effective.

[Wky2-2.2] How does RSVC avoid confounders?

There are two ways that confounders can be introduced: (1) Through visual features that co-occur in the images. For example, a beak and an eye will appear together in most images of birds or (2) the concept extraction method entangles two visual features even if the network encodes them independently. To address (1), we make use of image patching which helps us break down correlated visual features. This does not guarantee that all visual features become uncorrelated. As suggested in the review, both RSVC and MCS would be impacted by this type of confounder. In the paper, we highlight that MMCS fails with respect to (2), the concept extraction process can lump together neurons that are actually encoding different things. L262: "However, correlation based similarity can be affected by confounds from concept extraction". In contrast, RSVC uses lasso regression directly from the activation matrix A1 and avoids the issue of confounds introduced by the decomposition of M1.

[Wky2-2.3] Replacement test basis?

RSVC learns a map from A2 to U1. This follows the intuition that, if a concept from M1 is encoded by M2, there should be a neuron or group of neurons in M2 that can recreate the concept coefficients from U1. The concept basis for M2 does not play a role in this mapping. Please do not hesitate to respond if we have not interpreted the question correctly.

[Wky2-3.1] Is RSVC also a surrogate model?

Thank you for pointing this out. After consideration, we believe that you are correct that technically RSVC also uses a surrogate model due to the NMF decomposition. We have reworded the sentence to remove the word surrogate, but still emphasize the critical advantage of our method, which is using the network's activations directly.

[Wky2-3.2] Linear predictors have been used in the disentangled representation learning literature.

Thank you for highlighting these methods, we have added them to the related works section.

[Wky2-3.3] Section 3.3 is fairly redundant.

We move the details of Section 3.3 to the appendix for space considerations.

Thank you for your review.

(8D1Q-1) What is the core goal of this work? Is it to present a tool?

We agree that one of the main contributions of this work is to present a tool for surfacing model differences. In this work, we contribute by (1) defining the problem of visualizing model differences, (2) providing a tool to solve the problem, and (3) establishing that the tool measures what we intend it to measure.

(8D1Q-2) Novelty and usefulness of the tool.

We have updated the manuscript to clarify the importance of this problem for the field of XAI. In general, XAI methods sacrifice model fidelity to produce explanations that are simple enough for human interpretation. While simplification is important, it’s not clear how much is appropriate. For example, it is possible that simplification leads to indistinguishable explanations for all models trained on the same data. On the other hand, preserving complexity leads to a large number of neurons or concepts to inspect, making it extremely challenging to understand why a model makes a particular decision and which features are actually interesting. Our work is novel as it is the first to take complex models, without any specific training to disentangle the representations, and show that XAI methods can be adapted to find features that are unique and important to that model. Our approach is useful as it allows users to filter complex explanations for interesting features that contribute meaningfully to behavior. We hope that, in the future, this can be used to give complete explanations for model failures, an unsolved challenge in XAI [R1].

[R1] Colin et al., I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods, NeurIPS 2022

(8D1Q-3) Value of Figures 3, 4, and 5.

We are pleased that you find insights from Figs 3, 4 and 5 interesting. We include these figures because they answer critical questions that assess the validity and usefulness of the proposed tool. Fig 3 shows that RSVC can discover concepts that are unique and important to a model, Fig 4 links representational differences in two model’s to specific changes in a model’s behavior, and Fig 5 (1) verifies existing observations in the literature (2) provides a different perspective into how layers relate in terms of concepts. As a small step in trying to gain deeper insights, we provide a new experiment in the Appendix A3 comparing a model trained on NABirds to a model trained on NABirds and Stanford Cars. We find that there are some consistent differences in the types of features learned. The model trained on both NABirds and Stanford Cars exhibits complex concepts that are semantically aligned with specific car models. For example, one discovered concept is sensitive to the racing stripe on a type of muscle car. In contrast, car concepts for the model trained solely on NABirds appear to be sensitive to a particular color and style of car, but are not specific to the car model. RSVC opens the door to future focused evaluation on a specific model and a small set of classes to present a complete picture of model behavior. In this work we have focused on a broader exploration of the tool's properties to demonstrate its potential.

(8D1Q-3) Discussion of additional prior works.

We included the suggested references in the latest version of the manuscript (Sec 2). Bau et. al PNAS 2020 is designed for single networks, to the best of our understanding. The authors visualize the activations of a convolutional neural network and use an object detector to decide how well the activations match a semantic concept. In order to compare two networks, the authors compare the statistics of the detected concepts, but this is limited by the quality of the detector. In addition, detectors are trained on datasets of known concepts, but networks can discover and use novel and abstract concepts. The work of Dravid et. al ICCV 2023 is most similar in spirit to our work. Their work correlates the activation patterns of neurons in two different networks. They seek out neurons with similar activation patterns and coin these neurons to be “Rosetta Neurons”. However, they do not aim to provide a method to test for dissimilar neurons that significantly contribute to model behavior, which is the primary focus of our work. In addition, Rosetta Neurons are found through correlation, which can miss many-to-one mappings from one model to the other. Finally, activation atlases (Carter et. al Distill 2019), are an excellent tool to visualize model concepts as a whole. The focus of their method is to develop a 2-D map of feature visualizations that is organized in a way that reflects the representation of the network. Different from RSVC, their work is not focused on quantifying and precisely measuring the differences between two models, rather they are seeking a broader, more general picture of network behavior. It would be interesting to develop an atlas that could be used for understanding model differences in the future. Due to these key differences between our work and the prior works, we believe that our work is not comparable. Thanks also for sharing the sparse autoencoders references, since our method operates using concept dictionaries, sparse autoencoders could be used instead of NMF in future explorations.

(8D1Q-4) Why patches?

We follow prior work (i.e. CRAFT) in using image patches because they help improve interpretability for the user. When using entire images, some visual features will nearly always co-occur. For example in the bluejay class, almost every image will contain the tail, body and beak. By splitting into patches we can identify that the model is specifically reacting to the tail. All models are trained with random resized crops, making the patches in-domain for the network. We have added a line to justify this choice in the text (Line 177-178).

Thanks again to the reviewers for their feedback. We have added a final revision of the paper where we add some of the additional analysis requested in the original reviews. Specifically, we include the following which can be found in the revised PDF:

• In response to RkXK and 73oU request we add parameter sweeps over the L1 parameter and the number of concepts (Fig. A12, Fig. A13). The new results show that increasing L1 penalties results in more zero coefficients and lower similarity values. Increasing the number of concepts decreases the reconstruction error, but also results in more low similarity concepts.

• wky2 and 73oU found that the volleyball concept was challenging to interpret. We have replaced the main figure with a clearer example. However, to better understand the volleyball concept, we follow fUV9’s suggestion and include a deeper analysis of the neurons in the RN18 used to predict the volleyball concept in RN50 (Fig. A14). This visualization uses permutation importance to identify the most important neurons for the regression model. The neurons demonstrate a sensitivity to a variety of visual features, including the boundaries between nets and the background, volleyballs in the air, players in gyms, and close-ups with nets with hands nearby. These neurons qualitatively seem to explain some of the over-predicted samples.