FaCT: Faithful Concept Traces for Explaining Neural Network Decisions

We thank the reviewer qa8G for their in-depth feedback and questions. We are very happy to see that the reviewer finds our approach generalizable, our model competitive and our metric novel. In the following, we will discuss the Weaknesses (W1-4), Questions (Q1-3), and Limitations (L1-2) raised by the reviewer.

Interpreting the Concepts (W1, L1): We intentionally avoided using pre-defined concept sets that would bias the interpretation towards what we as humans expect the model should learn, and instead used an unsupervised concept-discovery method (SAE) for identifying the concepts the model finds useful. With FaCT, we aim at making the concepts that the network learned for decision-making transparent, which do not need to align with pre-defined human concepts. This allows us to maintain the performance of powerful models while allowing to interpret their reasoning. That said, we agree that having explicit names assigned to the concepts can benefit the interpretability for the end user, and have therefore followed this suggestion and used CLIP-Dissect [R1], which labels neurons (SAE latents in our case) using a CLIP model. Applying CLIP-Dissect on our concepts to generate the names would name the the concepts (A-F) shown in Fig. 6 to the following words (for each concept we show the top-3 matches):

[A]: volleyball, rugby, balls

[B]: jerseys, uniforms, players

[C]: rugby, sergio, concentration

[D]: gymnastics, acrobat, flexible

[E]: volleyball, tournaments, serve

[F]: basketball, layup, contested

We find these names to be quite consistent with how we would interpret these concepts. We will add this discussion to the final revision.

Using a pre-trained network (W2, Q1): We thank the reviewer for raising this point. There are two main reasons why we focused on training FaCT with pre-trained B-cos checkpoints.

First, our current two-step procedure for training FaCT allows it to be readily applied to existing trained B-cos models, which is particularly important given the recent focus on extending B-cos networks to different domains such as Natural Language Processing [R2] and Self-supervised models [R5].

Second, SAEs have been mainly proposed as an unsupervised concept-discovery method to be applied on top of a structured (trained) representation, with the sparsity constraints being the only prior guiding the model towards identifying concepts (i.e clusters). We thus suspect that training them on top of an unstructured representation may add to their instability or lead to less-interpretable latents.

We do however agree that having the choice of one-stage training from scratch can be desirable in certain cases, and will add this in the future work discussion.

Other Datasets (W3, L2): We thank the reviewer for their suggestion. We mainly experimented with ImageNet to ensure the competence of FaCT. Following your suggestion, we evaluated FaCT also on the (not-cropped) CUB dataset for ResNet-34 architecture. Similar to ImageNet experiments, we started with a pretrained B-cos checkpoint (ImageNet B-cos finetuned on CUB) and trained our SAEs at different layers, having the rest of the model frozen. The results can be seen in the table below.

Table 5: Performance for ResNet34 on uncropped CUB dataset

(Standard: 76.0 from [13])

We observe that FaCT works out of the box on smaller-scale datasets such as CUB and is able to maintain the accuracy (<0.8 % drop across). We thank the reviewer for their suggestion and will add the CUB experiments, together with qualitative results of the shared concepts for the final version.

Comparison to ProtoPNets and CBMs (W4): As our main focus was large-scale ImageNet evaluations, we mainly compared with models that extend to ImageNet, such as B-cos channels and post-hoc methods such as CRAFT and CRP. Existing implementations of Part-Prototype often do not extend to ImageNet. In the two tables below, we compare a set of ProtoPNets and CBMs that had a comparable setup to ours.

Table 6: ResNet34 comparison on CUB and ImageNet

*Numbers from [13], trained on uncropped CUB images ; **Numbers from [46]

Table 7: ResNet50 comparison on ImageNet

***Numbers from [34], using Torchvision V1 recipe

Besides the significant performance advantage on CUB and ImageNet, FaCT offers a shared concept-basis, extends to different layers, and architectures (CNNs and ViTs), all while providing a faithful input-level attribution of concepts.

Performance Margin to Vanilla Models (Q2): We would like to highlight that the recent V2 version of B-cos models [9] that we use obtain much closer performance to vanilla models. (e.g. only 0.8% drop on DenseNet-121). We would like to point to Table 5, 6, and 7 above. In all the three tables we observe that FaCT models are performing on par to standard baseline (Maximum 3.3% drop across all tables). We also observe that on a smaller-scale (CUB) dataset presented in Table 6, the performance gap compared to B-cos is even smaller (<0.8%). In the revision, we will add the vanilla baseline numbers to the comparisons for clarity.

Random Baseline in C2-Score (Q3): We observed consistent results for the random baseline across runs. In particular, for multi-class evaluations, the consistency score of the random baseline was at 0.004 ($\sigma$=0.0005), averaged across three seeds. For the per-class setup, the random baseline score was 0.13 ($\sigma$=0.021), averaged across the random baselines of all 1000 classes. We will add these for the final revision.

[R1]: CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks, ICLR 2023

[R2]: B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainability, preprint Feb 2025

[R5]: How to Probe: Simple Yet Effective Techniques for Improving Post-hoc Explanations, ICLR 2024

We would like to begin by thanking reviewer RaLA for their detailed feedback and questions. We are happy to see the reviewer finds our idea novel, our method generalizable, and our writing easy to follow. In the following, we will address the Weaknesses (W1-3), Questions (Q1-3) raised by the reviewer.

Human-alignment and Scalable Labeling (W1): Please note that there is an inherent difference between explicit human-aligned models, such as CBMs, that are forced to learn a mapping to pre-defined concepts and conventional models, that learn concepts that they deem useful, which need not correspond to what we as humans would like to see or even comprehend. The advantage of the latter is obvious in terms of performance, achieving much higher accuracy than e.g. CBMs at the cost of being less transparent. With FaCT, we aim at making the concepts that the network learned for decision-making transparent, which do not need to align with pre-defined human concepts. This allows us to maintain the performance of powerful models while allowing to interpret their reasoning. Here, we believe a visualization of concepts will provide a better understanding of what is happening than text, given that we work on a vision task, which was also a key motivation for ProtoPNet, PipNet and others. Nonetheless, if textual interpretations are preferred, we can use off-the-shelf methods to label our concepts such as CLIP-Dissect [R1] which name individual neurons using a language-aligned model (i.e CLIP). To satisfy our curiosity, we applied CLIP-Dissect [R1] to our SAE latents, and got the following names for the concepts (A-F) shown in Fig. 6 of the paper: (for each concept we show the top-3 matches):

[A]: volleyball, rugby, balls

[B]: jerseys, uniforms, players

[C]: rugby, sergio, concentration

[D]: gymnastics, acrobat, flexible

[E]: volleyball, tournaments, serve

[F]: basketball, layup, contested

These human-understandable textual names seem meaningful with respect to the concepts in Fig. 6. While not our primary focus, we will add this downstream application for textual explanations to our paper, thank you for raising this point.

Evaluation on other Datasets (W2): We primarily avoided using fixed human annotations to evaluate our shared concept basis as the annotations are class-specific, subject to arbitrarily chosen granularity level, and are independent of what a model might learn to be useful for a decision. We would like to refer to Fig. C1 in Appendix, where we show sample concepts that are not considered in Part-ImageNet annotations, while even the PCA visualization of DiNOv2 features seem to be better suited. We will make this comparison and discussion more prominent for the final version. We have also extended our evaluation to the CUB dataset, to ensure that our findings also hold on smaller-scale, but fine-grained classification (kindly refer to Table 3 in response to reviewer Lkv1).

Ablation on Hyper-parameters of SAE (W3): We have described our hyper-parameter sweep and model selection criteria in (Appendix A, Lines 669-684). We have also plotted the models in terms of sparsity vs. accuracy in Fig. B1, where each point corresponds to a particular (top-k, expansion factor, and learning-rate) configuration. We had deferred these results to the Appendix in lack of space. We will make the references more prominent for the revision.

Sensitivity to Hyper-parameters of SAE (Q1): In general for the same layer we observed many concepts being repeated across configurations as well as across architectures at similar depth. Following your question, we evaluated the recently proposed Stability Score [R4] for the same layer and dictionary size, but with different (top-k) sparsity factor. We did this both for early and late-layer decompositions of Densenet-121. While being trained on different models, datasets, and layers, our stability scores are quite on par with the ones in Table 1 of [R4], where the authors report 0.5 for the TopK-SAE under different seeds. We would also like to point out that FaCT would directly benefit from new variants of SAEs, e.g. [R4], that may be introduced by the community. We will extend the quantitative stability evaluation for the final revision.

Pair-wise Stability Score

Averaged over three pairs of experiments (TopK ∈ 8, 16, 32)

Applying FaCT to other Modalities (Q2): The main components of FaCT are the use of B-cos layers (for faithful input-attribution and output-contributions) together with SAE for interpretable concept-based representation. Both of these have been recently explored for other domains such as natural-language processing. In particular, B-cos LM [R2], introduces a B-cos language encoder and evaluates the interpretability and faithfulness of attributions on the input sequences. Additionally SAEs [6, 11] were originally introduced for language models. We thus believe FaCT would lend itself well for other modalities and will add this to the future work discussion for the revision.

Concept-Editing (Q3): Thank you for this interesting suggestion. Indeed one of the advantages of FaCT is that it uses concepts as part of the decision-making process, and hence modification to the concept representation can be explored and removing concepts does indeed affect the output behaviour of the model (e.g. Fig. 2). While B-cos models have shown to lend themselves well for being guided through their attributions, e.g. to avoid relying of spurious features (see Fig. 1 in [R3]), FaCT directly allows for steering the model on a concept-level (e.g. by explicitly masking certain SAE latents). We will add this as a future work direction in our revision.

We will also move the limitations section to the main text in our revision.

[R1]: CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks, ICLR 2023

[R2]: B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainability, preprint Feb 2025

[R3]: Studying How to Efficiently and Effectively Guide Models with Explanations, ICCV 2023

[R4]: Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models, ICML 2025

We thank the reviewer Lkv1 for their detailed feedback. We are happy to see that the reviewer finds our approach and our evaluation novel and comprehensive. In the following, we will further address the Weaknesses (W1-4), Questions (Q1) raised by the reviewer.

Human-alignment of Concepts (W1 & Q1): There is an inherent difference between human-aligned models, such as CBMs, that are forced to learn a mapping to pre-defined (human) concepts and conventional models, that learn concepts that they deem useful, which do not need to correspond to what we as humans would like to see or even comprehend. The advantage of the latter is obvious in terms of performance, achieving much higher accuracy than e.g. CBMs at the cost of being less transparent. With FaCT, we aim at making the concepts that the network learned for decision-making transparent, which do not need to align with pre-defined human concepts. This allows us to maintain the performance of powerful models while allowing to interpret their reasoning. Here, a visualization of concepts will give a much better understanding of what is happening than text, given that we work on a vision task, which was also a key motivation for ProtoPNet, PipNet and others. Regardless, we can use off-the-shelf methods to label our concepts such as CLIP-Dissect [R1] which name individual neurons using a language-aligned model (i.e CLIP). Following your suggestion, we thus applied CLIP-Dissect [R1] to our SAE latents, and got the following names for the concepts (A-F) shown in Fig. 6 of the paper: (for each concept we show the top-3 matches):

[A]: volleyball, rugby, balls

[B]: jerseys, uniforms, players

[C]: rugby, sergio, concentration

[D]: gymnastics, acrobat, flexible

[E]: volleyball, tournaments, serve

[F]: basketball, layup, contested

We find these names to be very consistent with how we would name the concepts in Fig. 6. We thank the reviewer for their suggestion and will happily add this discussion for the camera-ready.

Use of DiNOv2 for C2-Score (W2): Indeed, output features of DiNOv2, or any other foundation model used within our C2-Score metric, should not be considered as the oracle or ground-truth semantic space. The underlying motivation of using DiNOv2 features is two fold: First, DiNOv2 has recently shown remarkable performance for similar tasks such as semantic correspondence [3, 52, 53], which makes it suitable for the context of evaluating consistency of concepts across images. Second, we intentionally steered away from evaluating the concepts with respect to human annotations, as the annotations are class-specific, subject to arbitrarily chosen granularity level, and are independent of what a model might learn to be useful for a decision. We have demonstrated this extensively in Fig C1 in Appendix, where we show sample concepts that are not considered in Part-ImageNet annotations, while even the PCA visualization of DiNOv2 features seem to be better suited. We will make this comparison more prominent for the final version. Appreciating that DiNO is not the ground truth, we had complemented our analysis with a human user study to evaluate found concepts. We will further highlight this complementarity of evaluations and their individual limitations in the discussion.

Extending to More Datasets (W3): We mainly experimented with ImageNet as it is a realistic dataset covering a wide range of different classes and diverse images, which many existing interpretability works such as ProtoPNet or PipNet fail to scale to. Following your suggestion, we evaluated FaCT also on the (uncropped) CUB dataset for ResNet-34 architecture. Similar to ImageNet experiments, we started with a pretrained B-cos checkpoint (ImageNet B-cos finetuned on CUB) and trained our SAEs at different layers, having the rest of the model frozen. The results can be seen in the table below.

Table 3: Performance for ResNet34 on uncropped CUB dataset

(Standard: 76.0 from [13])

We observe that FaCT works out of the box on smaller-scale datasets such as CUB and is able to maintain the accuracy (<0.8 % drop across), while training much faster than e.g. ProtoPNets. We will add the CUB results together with qualitative visualizations for the revision.

Discussing Computational Overhead (W4): Following your suggestion, we measured the inference time over the entire ImageNet validation (50,000 Images). We did this for DenseNet-121 architecture on an A40 GPUs and averaged the results across three runs.

Table 4: Time to process ImageNet’s Test set on DenseNet-121 (average across three runs)

We observe only a negligible increase in inference time, which is likely to reduce further with inference optimization (e.g. changing the SAE-hooks to fixed layers and using torch.compile). We will add these results for the final version.

[R1]: CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks, ICLR 2023

We thank the reviewer V8ET for their constructive feedback. We are glad that they found the variety of our experiments and evaluation setup as a strength. In the following, we will further address the Weaknesses (W1-3), Questions (Q1-6), and Limitations (L1-4) raised.

Faithfulness of input-attributions (L1 & Q2): One main limitation of existing concept-based methods (or similarly part-prototype networks) is that they struggle with correctly attributing the activation of a concept (or prototype) at late-layer to the image. These attributions are usually approximated with up-sampling similarity maps from the late-layer [10, 17, 46] or post-hoc attribution methods [1, 48] and have shown to be unfaithful visualizations [23, 32, 48]. In addition, such approximations are even worse in ViT architectures, since the inductive (locality) bias of CNNs no longer holds. B-cos networks, however, provide model-inherent attributions that are faithful to the model in that they are the actual model’s computation. Precisely, for every input sample, they express the activation of the logit (or in our case concepts) as a linear mapping of the input, which summarizes the computation (See Eq. 3 for B-cos networks’s logits and Eq. 12 for FaCT’s concepts). This is independent of the layer of choice or the architecture. On top of the theoretical grounds, B-cos attributions have been shown to outperform other (approximate) attributions in terms of localization (see e.g. Fig. 6 in [9]).

By using B-cos transforms for layers in FaCT, we are able to retain all of these benefits when it comes to visualizing concept attributions allowing us to precisely demonstrate where in the image the concept is being activated for. We would like to highlight this further by referring to Fig. E2 and E3 in Appendix E. Notice that it is only with such high-resolution faithful attributions at input-level that one can retrieve the correct meaning of each of the shown concepts, e.g. ‘Red Dots’.

Different baselines across experiments (W2): Not all baselines make sense to compare in the context of every metric, as the metrics are task-specific. We elaborate below:

a) Accuracy (Fig 4): Post-hoc concept-based methods such as CRAFT do not directly use concepts for prediction, and hence one cannot measure “accuracy” for such methods. As a result, we compare against the B-cos model as a baseline, which is close to the accuracy of conventional models (see Table 1 below). For CRP the accuracy would correspond to the accuracy of the standard model, which as per your suggestion in (L2), we will add for easier reference.

b) Concept consistency (C2-score) (Fig 3): We evaluate this against all baselines, i.e. against channels of the B-cos model, as well as post-hoc methods such as CRAFT and CRP.

c) Concept Deletion (Fig. 2): This metric measures the accuracy of the importance value assigned to each concept, given a fixed set of concepts. Since each concept extraction method (FaCT, CRAFT, CRP, etc.) yields a distinct concept set, a direct comparison between them is not possible on this metric. As a result, we focus on comparing our approach of using B-cos attributions against alternative concept attribution methods used with these baselines, e.g. Sobol indices used in CRAFT and Saliency used in TCAV and VCC, and find that B-cos attributions achieve the sharpest drop, showing their effectiveness.

d) User Study (Fig. 5): The goal of this experiment was to complement the C2-Score gains observed for SAE latents compared to B-cos channels, under the same visualizations, with the same architecture, weights, layer, with both having a shared concept basis. . In addition, since we were working with sampled concepts, we opted for having more samples rather than many methods with very few samples from each.

User Study (Q3 & L3): To the best of our knowledge, there is no pre-defined protocol for evaluating fine-grained visual concepts. We have put our emphasis on developing a fair and unbiased study by introducing a double-blind control scenario, where we randomly distributed the users to 10 groups, each with randomly ordered samples from both methods. Similarly, when studying the effectiveness of visualizations, we performed counter-balanced AB/BA testing, where top-activating images of each concept were shown with and without the input-attribution in two different groups (see Fig. D2 in the Appendix). Regarding the definition of an interpretable concept with `meaining’, as this is abstractly defined in the literature, we followed the common approach of providing the user with examples. In particular, we provided examples of low-level, high-level, object-centric, and non-object-centric concepts to the users (see Fig. D1).

Vanilla Baseline (L2): We will add the standard baselines for the final version for easier comparison. For convenience, we have put a comparison table below, where we compare FaCT, B-cos, and the vanilla architecture.

Table 1: Comparison to Vanilla Baseline

* Trained with torchvision’s v1 recipe, as v2 is not available for DenseNets. (Other rows are trained with the v2 recipe.)

We observe a significant increase in the consistency (i.e C2-Score) of concepts, while remaining on-par on ImageNet accuracy across architectures.

Concept-Deletion Curves (Q1): We were indeed also curious about such a sharp drop for early layers (Block 2/4) seen in Fig. 2, as well as Fig. F1 in Appendix. For these early-layer experiments, we investigated the first few concepts that are removed, and found a very small subset of latents that occur on every image. This was primarily the case for early-layer experiments. In the case of Fig. 2, there are exactly 4 concepts that occur on 100% of the samples. We had also identified these while training the SAEs, discussed it in Appendix A (lines 675-682), and found similar phenomena in the existing literature (see points with frequency of 1 in Figure 3 of [31]). We hypothesize that these are mean feature vectors that are used for reconstruction in every sample. It is the removal of these latents that results in the sharp drops for early-blocks in Fig. 2. Nevertheless, as we use the same concept-set for evaluating the concept-importance methods, this still shows that B-cos is best in identifying the most impactful concepts.

To investigate the identification of most relevant concepts beyond these “always-on” shortcut concepts, we now investigated concept deletion when not allowing removal of these four latents and evaluated on the rest of the concept. We plotted the results similar to Fig.2 and found B-cos contributions to again consistently out-perform other attribution methods on concepts. Since we cannot upload images for the rebuttal, we report the Area Under the Deletion-Curve here in Table 2: (Lower means faster drop, hence better).

Table 2: Area under Deletion Curve for Block 2/4 on DenseNet-121

We observe that the sharp drop disappears once we do not allow the removal of these four concepts for any method, yet we observe that B-cos contributions still significantly outperform other concept-importance measures. We thank the reviewer for raising this point. We will make the discussion around always-occurring concepts at early layers (lines 675-682 in Appendix A) more prominent for the final version, while also adding this new experiment which excludes removal of these concepts.

Notation in Eq. 14 (Q3.5): Thanks for pointing this out. Indeed, in Eq (14), $f(I)$ should be replaced by $H(I)$, which refers to the foundation model’s features. We will correct this for the revision.

Random Concept in C2-Score (Q4): The random concept activates on every image and assigns random importance to each pixel (sampled uniformly from within [0, 1]). In a multi-class setup, the consistency (Eq. 16) of random concept was at 0.004 ($\sigma$ 0.0005), averaged across three seeds, while on a per-class setup it was 0.13 ($\sigma$ 0.021), averaged across the 1000 classes. This agrees with the original assumption that when the evaluation set $\mathcal{D}$ is restricted to a single category, all the output features of DiNOv2 become more similar. This motivated us to define C2-score as the difference w.r.t the random baseline.

Spatial Extent (Q5): The spatial extent is the average number of pixels required to cover 80% of the total positive attribution of a concept. This was defined so that we can demonstrate the variety of concepts, as opposed to prior work that often assumes concepts to fit in small patches [17, 33]. We have defined the spatial extent in Appendix E (Lines 786-792) and will make sure to make it more prominent in the revision.

Spatial Extent of ViTs (Q6): We believe the similarity in spatial extent across two layers is due to the full-image receptive field at all the layers of ViT. Following your suggestion, we will run an experiment at a very early-layer and extend the plot for the revision.

Clarity and Notation (W1, W3, L4): We will take all of the questions raised by the reviewer into account and revise for clarity. Regarding the notation, while we were trying to closely follow the notation in B-cos paper [9], we will make sure to adjust it to be more intuitive. We will also highlight references for the Appendix sections more clearly. We would be grateful if the reviewer could specifically mention what they find missing so that we revise for the revision.