Navigating Neural Space: Revisiting Concept Activation Vectors to Overcome Directional Divergence

Thank you for your constructive and insightful comments, as well as the positive feedback regarding the writing quality and our experiments. Below, we address your concerns and questions.

• (W1) Table 1 / significance of results: While we do see large improvements in artifact relevance for the real artifact (band-aid in ISIC2019) for VGG16 (31% vs 46%) and ResNet (22% vs 26%), it is true that the artifact relevance remains consistent (22%) for EfficientNetB0. However, $\Delta\text{TCAV}^{\text{gt}}$ largely improves (0.03 vs 0.11), indicating that the reliance on the abstract concept “band-aid” is highly reduced. In addition, the heatmaps in Fig. 6 confirm the reduction in artifact relevance through Pattern-CAVs for various real-world artifacts (band-aid, ruler, skin marker) for VGG16. We also want to stress that the improvement of bias mitigation approaches, such as RR-ClArC, is subject to future research to further decrease the artifact relevance. However, this is out of the scope of this paper, as our focus is the improvement of the precision of concept representations. Our findings indicate that Pattern-CAVs clearly address the shortcomings from Filter-CAVs, leading to precise concept directions. We would also like to stress that we do not regard the Pattern-CAV and Filter-CAV necessarily as competing methods, but rather as two complementing approaches providing two fundamentally different views on the classification problem. When looking at the problem through the lense of the “filter”, we aim to optimally separate the data (“discriminative” perspective), e.g., in neuroimaging separate patients from healthy controls. When looking at the problem through the lens of the “pattern”, we aim to identify, e.g., the brain areas where the two populations differ (“explanatory” perspective). The impactful insight of Haufe et al. (2014) was that these two views are not the same. Thus, when precise concept directions matter (e.g., concept erasure/addition, TCAV), we recommend the usage of Pattern-CAVs, due to the addressed shortcomings of filters. However, when a good decision boundary matters to predict the existence of a concept (e.g., as in concept-bottleneck models), Filter-CAVs are better suited. We strongly believe that these two perspectives are not known to the community of CAV users.
• (W2) Grayscale images: We would like to kindly refer to our experiments with grayscale images, specifically the Bone Age dataset, in which we manipulate the brightness to insert a controlled artifact (see Fig. 9 in the appendix). In our experiments, (1) we measure a high alignment between Pattern-CAV and the ground truth concept direction (see Fig. 3, right) and (2) successfully decrease the reliance on the artifact by leveraging the CAV-based bias mitigation approach RR-ClArC (see Tab. 1, first value per cell). This confirms the superiority of Pattern-CAVs over Filter-CAVs also for grayscale images.

Thank you again for your very insightful comments and the interesting discussion!

• How do we ensure the model picks up brightness artifact?: It is true that the bone age dataset already exhibits a wide range of brightness levels. However, our experiment does not require manipulated images (i.e., images with artificially increased brightness) to be distinct from other, original images with a high brightness level. Assuming an approximately uniform distribution of brightness level over classes before our attack, the manipulation of a subset of images of exactly one class shifts that distribution. After the attack, there is a clear correlation between brightness level and class label, even if other classes contain a few images with an original brightness level that’s indistinguishable from our attacked images. This correlation can be picked up by the model as a shortcut. In addition, it is to note that during training, we do not explicitly force the model to use the shortcut. However, after training, we can measure that the model indeed picked up the brightness artifact as a shortcut. Specifically, the accuracy of the Vanilla model on the biased dataset is significantly lower than on the clean dataset. For example, for VGG16, the accuracy drops from 78% on clean data to 50% on attacked data, confirming that the model uses the brightness manipulation as a shortcut.
• Other grayscale artifacts in bone age?: There are indeed other interesting artifacts in the bone age dataset. Based on your valuable suggestion, we ran two additional experiments using the bone age datasets with (1) an artificial grayscale timestamp artifact and (2) the “L”-marker as a naturally occurring artifact. For the latter, we follow [1], who found that models pick up the fact that “L” markings are larger for hands of younger children, as images are scaled such that all hands are of similar size. As a consequence, images of smaller hands (i.e., younger children) need to be enlarged more, leading to larger “L”-markers. Results for VGG16 are provided below and summarized in the following. We further updated the appendix with results for VGG16, ResNet18/50, EfficientNet-B0 and EfficientNet-V2 (see Sec. D.8, Tab. 10).
  • Grayscale timestamp: Pattern-CAV outperforms filter-based CAVs both in terms of accuracy on the biased test set and artifact relevance by a large margin across all model architectures, while maintaining a high accuracy on the clean test set. For example, for VGG16, the Vanilla model achieves an accuracy of 33% on the biased test set, the best filter-CAV 37% and pattern-CAV 70%. In terms of artifact relevance, the Vanilla model archives 72%, the best filter-CAV 59% and the Pattern-CAV 30% (lower is better).
  • Real artifacts (L-marker): For the real-world artifact, we report the relevance put onto the “L”-marker and the accuracy on the original (clean) dataset. The Pattern-CAV consistently reduces the artifact relevance the most, as shown below for VGG16. Here, the artifact relevance for the Vanilla model is 72% and can be reduced to 59% with a Filter-CAV and 30% with Pattern-CAV.

These experiments confirm the applicability of Pattern-CAVs for bias mitigation approaches, even for grayscale images, such as the Bone Age dataset.

VGG16 Results for (timestamp $\|$ “L”-marker)

[1] Pahde, Frederik, et al. "Reveal to revise: An explainable ai life cycle for iterative bias correction of deep models." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023.

Thank you for your constructive and insightful comments, as well as the positive feedback regarding our experiments. Below, we address your concerns and questions.

• (W1) Organization of paper: We decided to organize the paper as follows: After the introduction and related work, we discuss the filter-pattern problem, the resulting shortcomings of widely-used Filter-CAVs, and introduce our method, the Pattern-CAV as a robust alternative. Afterward, we demonstrate the effectiveness of our approach by (1) measuring the alignment of CAVs with true directions, and (2) evaluating their applicability in two applications, namely TCAV and ClArC. However, both applications are not part of our method, but means for evaluation. Therefore, in order to avoid confusion, we decided not to introduce TCAV in the methods-section.
• (W2) Novelty: We can assure that our work is the first to analyze, evaluate, and mitigate the shortcomings of CAVs stemming from the pattern-filter problem. A previous, unpublished version of this paper is available on arXiv (due to double blindness we can not provide the reference here), but the concept of pattern-CAV has not been introduced in a published paper before. Since filter-based CAVs are highly popular in today’s research and are used in hundreds of papers, e.g., they are the main basis of methods such as TCAV [1], ClArC [2], or in mechanistic interpretability [3,4], we believe that making the community aware of their fundamental shortcomings and proposing an improved pattern-based CAV variant is a novel contribution with a large practical value. While previous work such as Haufe et al. (2014) have shown that weights of linear models are susceptible to distractor components in multivariate neuroimaging and Kindermans et al. (2018) use these findings to introduce a new local explainability method (PatternAttribution), we are the first to report these shortcomings in extensive experiments in the context of CAVs. After a decade has passed, one can say without doubts that the results of Haufe et al. have had a large impact on the neuroimaging community. For instance, no one uses the filter weights of linear classifiers for the interpretation of EEG signals anymore, e.g., in the context of Brain-Computer Interfacing. On the contrary, it is unanimously clear that the corresponding pattern is the quantity of interest. We strongly believe that also the steadily growing community of CAV users will benefit from learning about the potential issues of linear classifiers for concept modeling, as well as the availability of a robust alternative.
• (W3) Realistic datasets: We want to stress that we also used real-world datasets (ISIC2019, Bone Age) with real-world, non-hand-crafted artifacts (ruler, band-aid, skin-marker) to solve highly relevant medical tasks. In addition to these real artifacts, we decided to insert controlled artifacts (timestamp, brightness) in separate experiments, to have full control over the artifacts for an improved evaluation. This allows (1) the computation of ground truth CAVs by the generation of pairs of images with and without concept (see next bullet point), (2) the computation of accuracies on clean and attacked test sets, and (3) the computation of artifact relevance via the existence of ground truth masks localizing the artifacts. In addition to evaluating our method on 3 different datasets with controlled and real-world artifacts, we will run experiments with additional datasets (ImageNet, CelebA) in the next few days and add the results to the paper.
• (Q1) Pairs of samples with/without concept: The controlled concepts/artifacts (timestamp, brightness) are inserted by manipulating the original image. Therefore, we can obtain a pair of samples with/without concept by loading the exact same image twice and manipulating only one of them. Examples for ISIC2019 (timestamp) and Bone Age (brightness) are shown in Fig. 9 in the appendix. In the case of FunnyBirds, the removal of a concept is implemented by the replacement of the concept (e.g., green wing of shape “wing-01”) by another, random design of the same body part, as visualized in Fig. 10 in the appendix.

Thank you for your constructive and insightful comments, as well as the positive feedback regarding the writing quality and our experimental design. Below, we address your concerns and questions.

• (W1) Related Work (Fel at al., 2023): We agree that [R1] is a highly relevant and related work and will mention it in our paper. In particular, we think it makes sense to discuss this work in the context of our experiments with unsupervised concept directions described in L266. While [R1] provides a holistic approach for the unsupervised detection of concept directions, our work leverages CAVs computed in supervised manner.
• (W2) Overlap with Dreyer et al. (2024): We can assure this is the first and original work to discuss and demonstrate the limitations of filter-based CAVs in the presence of noise and introduce Pattern-CAVs as robust alternative. Dreyer at al.’s contribution and main focus is the right reason gradient penalization method “RR-ClArC”, not the filter-pattern CAV problem. They do not claim to introduce Pattern-CAV, but rather refer to a previous, unpublished version of this work. Such an overlap in time, where a method is applied before the original paper introducing the method is published, is not uncommon in today’s research.
• (W3/Q1) Evaluation on additional datasets: We evaluate our method on 3 different datasets (ISIC2019, Bone Age, FunnyBirds) with controlled and real-world artifacts. We strongly believe these experiments are sufficiently demonstrating the benefits of our approach. However, we will run experiments with additional datasets (ImageNet, CelebA) in the next few days and add the results to the paper.
• (Q2) Statistical significance for results in Tab. 1: In addition to the standard errors already reported in Tab. 8, we will also report statistical significance using z-tests, both in Tab. 1 (main paper) and Tab. 8 (appendix). Thank you for the suggestion!

We hope our responses are satisfactory. Please let us know if anything remains unclear.

Best regards,

The authors

Thank you for your constructive and insightful comments, as well as the positive feedback regarding our experiments. Below, we address your concerns and questions.

• (W1) Novelty: We can assure that our work is the first to analyze, evaluate, and mitigate the shortcomings of CAVs stemming from the pattern-filter problem. A previous, unpublished version of this paper is available on arXiv (due to double blindness we can not provide the reference here), but the concept of pattern-CAV has not been introduced in a published paper before. Since filter-based CAVs are highly popular in today’s research and are used in hundreds of papers, e.g., they are the main basis of methods such as TCAV [1], ClArC [2], or in mechanistic interpretability [3,4], we believe that making the community aware of their fundamental shortcomings and proposing an improved pattern-based CAV variant is a novel contribution with a large practical value. While previous work such as Haufe et al. (2014) have shown that weights of linear models are susceptible to distractor components in multivariate neuroimaging and Kindermans et al. (2018) use these findings to introduce a new local explainability method (PatternAttribution), we are the first to report these shortcomings in extensive experiments in the context of CAVs. After a decade has passed, one can say without doubts that the results of Haufe et al. have had a large impact on the neuroimaging community. For instance, no one uses the filter weights of linear classifiers for the interpretation of EEG signals anymore, e.g., in the context of Brain-Computer Interfacing. On the contrary, it is unanimously clear that the corresponding pattern is the quantity of interest. We strongly believe that also the steadily growing community of CAV users will benefit from learning about the potential issues of linear classifiers for concept modeling, as well as the availability of a robust alternative.
• (W2) Lacks boundary optimization: We strongly believe that due to the aforementioned filter-pattern problem, optimizing for class-separability (or boundary optimization) is the incorrect optimization target to obtain precise concept directions. This is confirmed by two toy experiments simulating ambiguous boundaries: First, in a 2D experiment in Fig. 7 (bottom row) in the appendix, we sample with different random seeds from two distributions with different means and high standard deviation, leading to overlapping distributions. In this experiment, Pattern-CAVs (green) are more robust than Filter-CAVs (magenta). In a second experiment (Section D.3.2 in appendix), we gradually increase the mislabeling rate of known artifact samples to obscure the decision boundary, and observe consistently higher concept alignment scores for Pattern-CAV compared to Filter-CAVs (see Fig. 29 in the appendix).
• (W3) When to use pattern-vs-filter: When precise concept directions matter (e.g., concept erasure/addition, TCAV), we recommend the usage of Pattern-CAVs, due to the addressed shortcomings of filters. However, when a good decision boundary matters to predict the existence of a concept (e.g., as in concept-bottleneck models), SVM-CAVs are better suited. Please note that the choice whether to consider the pattern or the filter is not an arbitrary choice, but a fundamentally different view on the classification problem. When looking at the problem through the lense of the “filter”, we aim to optimally separate the data (“discriminative” perspective), e.g., in neuroimaging separate patients from healthy controls. When looking at the problem through the lens of the “pattern”, we aim to identify, e.g., the brain areas where the two populations differ (“explanatory” perspective). The impactful insight of Haufe et al. (2014) was that these two views are not the same. We strongly believe that these two perspectives are not known to the community of CAV users. We will make this guidance clearer in the paper.