Representational Difference Explanations

We thank mmBM for their comments and suggestions. Based on the requests, we have added new quantitative comparisons to further demonstrate the effectiveness of our RDX approach.

(mmBM-1) Baselines are relatively simple. Can we compare to more recent baselines?
In the paper we compare to a range of distinct baselines which span different families of methods, e.g., PCA, dictionary learning, and sparse autoencoders (see Fig. 3). They have been used with some success in interpretability tasks (e.g., in [D, E]) and we felt it was important to assess them on the model comparison task in this work.

As indicated in the review, there have been some recent contemporaneous works comparing model representations, e.g., [A] on arXiv in Feb 2025, [B] on arXiv in Dec 2024, and [C] on arXiv in Mar 2025. We already discuss [A, C] in our related works section (see L72 and L70). We found several challenges when trying to fairly compare to USAEs [A], so we decided to exclude it.

• The official code for USAEs has not yet been published at the time of this rebuttal. The USAE submission on OpenReview has a link to a GitHub page, but this leads to a 404 page. Also, the overcomplete repository (a Vision-based SAE Toolbox on GitHub) appears when searching for the USAE code, but this is because the README states that overcomplete was used by the USAE authors. Despite this, the code for training USAEs was not available in overcomplete.
• The methodology proposed by the authors of [A] requires training a universal representation space for every comparison. We had concerns about fairly evaluating their methodology due to the number and variety of comparisons in this work. For example, we conduct comparisons for several model variants trained on modified MNIST datasets, PCBM models trained on CUB, models trained on ImageNet, and models trained on iNaturalist. It is not clear if USAEs will be sensitive to dataset size and/or the number of models used in learning the universal latent space.
• Additionally, we work with small focused datasets with similar classes, it has not been established how to correctly apply USAEs to small scale settings like the modified MNIST experiment we present in Fig. 2.
• We also would also like to highlight Fig. A2, which shows several issues that are true of any linear decomposition method that make them less ideal for comparison tasks.
• Finally, RDX is explicitly designed to find differences, whereas both USAEs [A] and RSVC [C] may find differences as a byproduct of their main goals.

Thank you for the suggestion to include a comparison to NLMCD [B]. We were able to run experiments with this approach on our unaligned ImageNet representations (i.e., DINO vs DINOv2) and iNaturalist experiments from Fig. 6 and Figs. A7-A10. Note that this is a total of 7 experiments. We had to adapt NLMCD to make it compatible with our evaluation protocol as it generates an arbitrary number of concepts for each representation and measures the similarity of concepts from each representation. For our experiments, we selected the k concepts that had the lowest similarity score as the representative concepts for differences and measured the BSR metric. We find that RDX outperforms NLMCD consistently on all 7 experiments. We provide first, second and third quartiles for the BSR metric.

[A] Thasarathan, et al. "Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment." arXiv:2502.03714 2025
[B] Vielhaben, Johanna, et al. "Beyond scalars: Concept-based alignment analysis in vision transformers." arXiv:2412.06639 2024
[C] Kondapaneni et al... "Representational Similarity via Interpretable Visual Concepts." ICLR 2025
[D] Fel, et al. "Craft: Concept recursive activation factorization for explainability." CVPR 2023
[E] Cunningham, et al. "Sparse autoencoders find highly interpretable features in language models." arXiv:2309.08600 2023

(mmBM-2) Scaling due to requirement for pairwise distance matrices.
Our intended setting for this approach is the fine-grained analysis of a smaller dataset. We believe this setting to be quite reasonable for scientists (such as astronomers or biologists) studying specific datasets with vision models. Scaling this approach would require a sub-sampling strategy using a sparse similarity matrix to find groups of images that are near each other in the representational space. Beyond constructing a large similarity matrix, we see no other conceptual issues associated with applying our method to larger datasets.

Additionally, we would like to highlight that it is not clear that analyzing models at scale results in a meaningful understanding of the underlying models, as seen in recent evaluation studies [F, G]. We believe our approach of focusing on smaller datasets makes it easier to assess the efficacy of a method and will be more fruitful in the long-run. However, we agree that this is an important limitation to highlight and we will discuss it in the revised text.

[F] Kantamneni et al. "Are sparse autoencoders useful? a case study in sparse probing." arXiv:2502.16681 2025
[G] Smith et al. "Negative results for SAEs on downstream tasks and deprioritising SAE research (GDM mech interp team progress update# 2).” 2025

(mmBM-3) Issue with low dimensional SAEs. Why not use top-K SAEs with a larger latent space?
We chose to use low-dimensional SAEs to have a fair, consistent, and straightforward evaluation for all methods. However, it is true that SAEs tend to work better with an overcomplete basis. Based on this suggestion, we trained a top-K SAE with 50 concepts on unaligned and aligned experiments on the ImageNet subsets and iNaturalist subsets. We selected the 5 concepts with the largest average weight as representative concepts for our evaluation.

We present the BSR results of the 7 “unknown” experiments here for both aligned and unaligned representations. Q1, Q2, and Q3 indicate the first quartile, median, and third quartile respectively. We found that top-K SAEs outperformed SAEs, but still underperformed RDX, especially when comparing aligned representations. Aligned representations generally have more subtle differences making it more challenging for methods that are not designed to seek out differences.

Also, we are happy to include additional NLMCD results on Aligned representations for the final version. We did not have time to run this experiment as it requires some extra modifications of the NLMCD code.

(mmBM-4) Assumption that shared local concepts imply similarity in the representation space.
The assumption our work makes is actually the inverse statement. We assume that similarity in the representational space implies that there is a shared concept in the input space. This is a commonly used assumption in concept discovery, eg., [D, E]. For example, in Craft [D], concepts are visualized via the maximally activating images for that concept. The images that fit this criteria are considered similar in the representational space, specifically along the direction of that concept.

By visualizing images that are considered similar by the model, we hope to identify a consistent visual feature that the model has picked up on. If the model organizes images by a subtle, small concept, then we assume it will be one of the consistent visual attributes in our explanation grids. We will clarify this assumption in the revised text.

(mmBM-5) Can dictionary learning methods uncover any different types of representational differences compared to RDX?
We found it challenging to interpret the results from the dictionary learning methods due to the issues we highlight in Fig. A2, i.e., even when there was a concept discovered by a method, it was unclear how unique it was and how to interpret it alongside the other concepts extracted by the method. In contrast, RDX identifies concepts that maximize the uniqueness of the concept and does not require reasoning over multiple concepts in a weighted linear combination. In Fig. 2, we show how RDX focuses on the core difference between representations and results in easy to interpret explanations.

(mmBM-6) Definition of visual concepts on L57.
We appreciate how this definition could lead to confusion. Generally there are two approaches to concept based XAI methods: (1) concepts are pre-defined via a set of inputs [H] and then the model is analyzed to see how much the concept contributes to decision making or (2) concepts are discovered directly from the model’s activations [D]. Our definition only applies to the second setting, we will adjust the definition to be more general. Thanks for the suggestion.

[H] Kim et al. "Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)." ICML 2018

We thank e4CM for their comments and suggestions. Please do not hesitate to follow up in the discussion period if there are any remaining questions or points that we can clarify.

(e4CM-1) How were the qualitative results in the paper selected?
We select the clearest and most representative examples for the main paper. We include the full set of explanations for most comparisons in the appendix. We selected two concepts to visualize in Fig. 6 due to space constraints, the remainder are available in Figs. A8 and A10. More results for MNIST are found in Fig. A4 and more results for the CUB experiments are shown in Figs. A5 and A6. Complete results for the Mittens (ImageNet), Primates (ImageNet), Maples (iNat), and Corvids (iNat) are available in Figs. A7-A10. There is no specific objective criteria for choosing which results are presented in the main text vs. presented in the appendix. However, we feel that we have presented a large number of results from our experiments for the reader to inspect.

(e4CM-2) BSR is cyclic. Is it possible to improve evaluation?
We designed BSR to measure a property of difference explanations that we believe is important for comparing representations, i.e., finding groups of inputs that are considered similar by only one of two representations. We show that existing methods do not succeed at this task and design a method, RDX, to optimize for this metric (see Figs. 2, 4, and 5). We agree that it is perhaps not surprising that our method performs better than existing methods for this task, but we report BSR as it demonstrates that we have succeeded in our goals. Additionally, we have tested BSR with different distance normalization schemes and found that RDX generally performs the best even when the distance normalization schemes are mismatched (Fig. A13).

We agree that an in-depth human evaluation is out of the scope of this work. We include many qualitative visualizations in both the main paper and appendix for the reader to scrutinize and assess if the explanations are meaningful. In addition, in Sec. B.3 we report results for some case studies to discuss how explanations may, or may not, link to classification performance.

To strengthen our quantitative results further, following suggestions from jwKi and mmBM, we include new metrics from [A] and a new method that was explicitly designed for comparison [B] in our results. The new metrics can be found in the response to jwKi (see response (jwKi-1)). The new method’s results can be found in the response to mmBM (see response (mmBM-1)).

[A] Dreyer, et al. "Mechanistic understanding and validation of large AI models with SemanticLens." arXiv:2501.05398 2025
[B] Vielhaben, et al. "Beyond scalars: Concept-based alignment analysis in vision transformers." arXiv:2412.06639 2024

(e4CM-3) Points may be nearby using Euclidean distance, but far apart on the data manifold. How is this handled?
This is a good point, two datapoints may indeed be close together based on Euclidean distance but quite distant if we follow the manifold that the data lies on. We do not explicitly handle this case, but there are two reasons we believe RDX can work despite this issue.

(1) Many models (e.g., like DINO) are trained to have meaningful Euclidean distances in their latent space and so we assume the distances are meaningful. For classification models, the last layer is a linear classifier, resulting in a linearly separable representation. While this does not guarantee meaningful Euclidean distances, it strongly implies it. We use the last layer for all experiments in this work.

(2) It is unlikely that the nearest neighbors for a given point have small Euclidean distances, but large distances on the manifold, since we generally expect that representations are locally linear. Neighborhood distances and our locally-biased difference function push RDX to seek out groups of points that are locally linear. Notably, the non-linear dimensionality reduction technique Isomap [C] uses top-k nearest neighbors to construct a graph and estimate distances along the manifold of the data. Our locally-biased difference function is a soft version of the same idea.

Our results are reported across a variety of backbones and training objectives (e.g., see Table A5) and demonstrate robustness in the presence of this variation. Finally, RDX is not restricted to Euclidean distances. In the future, it would be interesting to explore other choices for the various components of the algorithm.

[C] Tenenbaum, et al.. "A global geometric framework for nonlinear dimensionality reduction." Science 2000

(e4CM-4) How does adding or removing points affect explanations?
RDX explanations are conditioned upon both the model and the input data. To the best of our knowledge, this is true for all concept-based explanation methods. For example, dictionary-learning approaches (NMF, SAE, PCA, etc…) will have different dictionaries depending on the data used to generate model activations. If the input data changes, the representation will change, and, subsequently the explanations will change.

However, if the number of points changes, but the data distribution stays the same, the concepts discovered will be very similar. We have conducted a new experiment, in which we add or remove 20% of the data (from the same distribution) and observed that the spectral clusters are highly similar. We are happy to include this result in the revised paper. Thanks for the suggestion!

(e4CM-5) How does representational alignment work? When does this step happen, how does it change the explanations?
The step does not happen for all experiments. For the “unknown” difference experiments (ImageNet and iNaturalist), we conduct experiments without and with alignment and report results for both unaligned (Fig. 3B Unaligned) and aligned representations (Fig. 3B Aligned). When aligning representations, we test alignment in both directions. We align representation A to representation B and vice versa. We use the symbol ‘ to indicate aligned representations (see L180). In Fig. 3B Aligned, we can see that results are reported twice for the baseline methods. KMeansA’ indicates that A was aligned to B, and KMeansB’ indicates that B was aligned to A. The qualitative results we show for these unknown experiments are all after alignment (Fig. 6 and Figs. A7-A10).

Representations may be misaligned due to a linear transformation of the representations. In these cases, aligning the representations before comparing them can help us isolate more fundamental differences (Sec 3.4). Alignment does change the explanations. In Fig. A3, we visualize the effect of alignment and show how it changes the explanations. We will highlight this result in the revised text and update the description of the alignment so that it is clearer.

We thank jwKi for their careful reading of the paper and insightful comments and suggestions.

(jwKi-1) Concerns about using only the BSR metric.
We agree that BSR is a desirable property for difference explanations and for this reason our design of RDX is optimized for the BSR target metric. We report performance using this metric as it best matches our methodological goals, i.e., finding groups of inputs that are considered similar by only one of two representations. Additionally, we report results for the BSR metric using alternative distance normalization methods in Fig. A13. There we observe that even when the normalization method used by RDX and the normalization method used in evaluation with BSR are mismatched, RDX still performs the best. This demonstrates RDX’s robustness even when not aligned directly with the evaluation metric.

Nonetheless, we agree that additional evaluation metrics would strengthen our results further. To address this we report additional results using the metrics from Dreyer, et al. [A] as suggested. These metrics use an independent, “semantic” model, OpenCLIP, to assess the clarity, redundancy, and polysemanticity of a concept via its explanation. We computed these metrics for the ImageNet and iNaturalist experiments (from Sec. 4.4 and Fig. 6) for both unaligned and aligned representations. Note that for redundancy, we adapt the metrics to compare the explanation grids from each model. Essentially, we compute the similarity for each concept from Model A to each concept from Model B. We use this similarity matrix to compute the redundancy metric (which summarizes the similarity matrix). We present the results below. Q1, Q2, and Q3 indicate first quartile, median, and third quartile, respectively.

Unaligned

The reduction in redundancy is even more apparent after alignment.

Aligned

We observe that RDX generates significantly less redundant explanations as we would expect. We do notice that the RDX explanations are slightly less clear than some other baseline methods according to some of these metrics. However, most methods have similar clarity and polysemanticity scores. We would like to emphasize three caveats to interpreting these results: (1) RDX generates explanations for differences, which naturally highlights parts of the representations that are more likely to be complex and less clear. (2) The metrics here presume that the generalist model (OpenCLIP) is an unbiased evaluator, but it is likely that OpenCLIP has its own biases that influences what it estimates is more or less clear/redundant. (3) Explanations for complex, niche datasets that are out of domain for OpenCLIP would likely have significant issues when evaluated using this metric. A benefit of the BSR metric is that it is insensitive to the complexity of the input images being analyzed.

Additionally, RDX is the only method in this work that is able to isolate and explain differences between two representations as seen in Figs. 2, 4, 5, 6, and A4-A10. We believe that a slight increase in concept complexity is natural and acceptable when focusing on understanding nuanced representational differences.

[A] Dreyer, et al. "Mechanistic understanding and validation of large AI models with SemanticLens." arXiv:2501.05398 2025

(jwKi-2) Scalability of RDX beyond pairwise comparisons.
This is an interesting question. There are at least two challenges that would need to be addressed. Suppose we have three representations, A, B, and C. (1) If we want to identify the differences between A vs. “B and C”, we have to decide if this means the union of the differences between (A, B) and (A, C), or the intersection. (2) We would need to develop a method to merge difference graphs from each pairwise comparison so that the output of clustering is the union/intersection of the concept sets. While this would scale combinatorially, RDX is fast enough on datasets of 5k points with a 512 dimensional latent space (32 seconds), that in some settings the pairwise comparisons would be feasible. Our work represents one of the first approaches to automatically identify explainable differences between representations from two different models. Extending our work efficiently beyond pairs of models, is a really interesting question which we leave for future work. We will highlight this opportunity in the revised text.

We thank HVHo for their feedback and suggestions. We believe that addressing these comments will make the paper even clearer.

(HVHo-1) Why conduct experiments on a modified MNIST?
In our first experiment, we compare two checkpoints trained on a MNIST dataset modified to include only the digits: 3, 5, and 8 (see L225). Our motivation was to create a simple and clean experimental test bed that could demonstrate an issue we observe with existing explanation methods like NMF, KMeans, PCA, SAE, etc. In Fig. 2, we show that existing methods, that are designed to analyze one representation at a time, are not suitable for comparing representations across models as they result in explanations that are unrelated to the core difference between the representations. In Fig. A2, we discuss issues with explanations that rely on linear decompositions of activation matrices as well. Importantly, we do conduct experiments on more realistic settings using modern models (DINO, DINOv2, CLIP, PCBMs) on real datasets (CUB, ImageNet, and iNaturalist). We will clarify the motivation why we selected this modified version of MNIST in the revised text.

(HVHo-2) Comparison with additional XAI methods.
We compare RDX to a variety of XAI methods (e.g., NMF, KMeans, PCA, and SAEs ) throughout the work (e.g., see Fig. 3). They are popular [A, B] both in practical applications and as the basis of recent research [D, E]. Following the suggestion from mmBM, we include comparisons to an additional explanation method (NLMCD) [C], designed for comparing representations specifically, and observe that our method outperforms it on the BSR metric. We test NLMCD on our unaligned representational comparisons for ImageNet and iNaturalist (Fig. 3B, Unaligned). The results are presented below.

[A] Fel, et al. "Craft: Concept recursive activation factorization for explainability." CVPR 2023
[B] Cunningham, et al. "Sparse autoencoders find highly interpretable features in language models." arXiv:2309.08600 2023
[C] Vielhaben, et al. "Beyond scalars: Concept-based alignment analysis in vision transformers." arXiv:2412.06639 2024
[D] Bussmann, Bart, Patrick Leask, and Neel Nanda. "Batchtopk sparse autoencoders." arXiv preprint arXiv:2412.06410 2024)
[E] Gao, Leo, et al. "Scaling and evaluating sparse autoencoders." arXiv preprint arXiv:2406.04093 2024

(HVHo-3) Analysis is largely visual, is there numerical evaluation?
In our opinion, the most effective way of explaining the behavior of interpretability methods for vision models to humans is via visual examples. Thus, we have used visual examples throughout the paper to demonstrate our results to the reader. However, we also report comparisons to existing methods using quantitative metrics. The primary goal of our work is to develop a method that can compare two models and reveal the concepts unique to each one. We measure this quantitatively using the Binary Success Rate (BSR) metric, described in Sec. 3.5. The results of this metric are presented in Fig. 3, in which we compare several methods on comparing models on multiple datasets. Ablation experiments can be found in Table A3. Additionally, following the suggestion from reviewer jwKi, we have included new quantitative metrics from [F] and find that RDX performs well under these metrics as well. The table and analysis is available in the response jwKi-1.

[F] Dreyer, et al. "Mechanistic understanding and validation of large AI models with SemanticLens." arXiv:2501.05398 2025

(HVHo-4) Tables are placed in the appendix making it hard to follow due to reliance on notation.
Thanks for the suggestion. We will polish the notation and clearly point to the additional results in the appendix.