Restyling Unsupervised Concept Based Interpretable Networks with Generative Models

Questions

• Q1: The concepts do not always precisely correspond to what we would consider as concepts from a human perspective. Some of these concepts can be entangled, especially if they are attending the same region of the image or object. We try to disentangle them through constraints like the orthogonality loss, but it is difficult to completely eliminate, even for supervised CoINs [A]. We discuss this limitation in Appendix H.

[A] M. Havasi, S. Parbhoo, F. Doshi-Velez. "Addressing Leakage in Concept Bottleneck Models". NeurIPS 2022

• Q2: We manually inferred the "name" of concepts from the changes that we observed. This is commonly done for naming visual concepts, e.g. with most activating samples (MAS) or in concept discovery (CRAFT [B], ACE [C], etc).

[B] T. Fel, et al. "Craft: Concept recursive activation factorization for explainability." CVPR 2023
[C] A. Ghorbani, et al. "Towards automatic concept-based explanations." NeurIPS 2019

Thank you for the review. We respond to your comments pointwise below:

Weaknesses

• W1: These are interesting points and related to the second limitation in Appendix H. Our visualization is limited by the quality of the generative model and its latent space. If a generator is not capable of representing a feature in its latent space, then we can't visualize it.

  An implicit assumption is that the generator is good at modelling input related features. If true, it is more reasonable to expect that the generator provides a rich latent space that represents a larger range of input features than needed for classification. In such a case the generator shouldn't induce additional bias for learning concepts than that of the classifier. The model will learn to map the concepts extracted by the classifier to latent space of the generator. This is why we require the generative model to be good at generating the given data distribution and finetune our $G$ before training VisCoIN when that is not the case.

• W2: We analyzed the impact of K in Appendix F.3. Smaller K leads to worse reconstruction (and accuracy), while higher K improves all metrics, since it allows for more expressivity in the concept representation. However, finding a good upper bound is subjective. We want a small dictionary for reduced overhead when interpreting the concepts, but a higher dictionary can improve disentanglement and reconstruction. In practice, we selected K=256 for CUB and Stanford Cars, and K=64 for CelebA. Our choice was influenced by (i) the number of classes, (ii) using number of concepts in supervised CoINs as a reference, and (iii) experiments showing that increasing K did not noticeably improve results.

• W3:

  • Evaluation novelty: We would like to clarify that we propose a novel strategy to evaluate faithfulness in the context of unsupervised CoINs, that uses the decoder/generator to explicitly modify the semantic content encoded by the concepts. We agree faithfulness metric is widely used for interpretability methods. In the main text too we initiate discussion about our faithfulness evaluation by recognizing the previous strategies to evaluate faithfulness (L380--384). We'll modify the text to make it explicit everywhere.

  • Details about consistency: We cover the details in Appendix C.3.1. We trained a linear SVM from the output of the second block of the ResNet-50 encoder, which are feature vectors of dimension 512 after pooling each feature map. We are currently implementing the additional consistency experiment, and hope to complete it within the discussion period, but it will be added in the revised version.

  • AUC metric for faithfulness: Thanks for the nice suggestion. We performed preliminary experiments comparing VisCoIN and FLINT on CUB-200 by adding most activated $N=4, 64, 128, 256 \text{(all)}$ concepts for each test sample to $\Phi(x)$ initalized to 0, and plotting the accuracy of g(.). We report the AUC below. Note that since the accuracy is on generated images, it is lower than accuracy of $g(.)$ on dataset. Method | AUC-FF metric | Accuracy of $g(.)$ on full reconstructions --- | --- | --- VisCoIN | 0.396 | 58% FLINT | 0.041 | 4.5%

    We'll repeat the experiments with greater resolution for $N$, but the results are already strongly in favor of VisCoIN. We expect VisCoIN to generally outperform other unsupervised CoINs on this metric as it's capable to generate high-quality reconstructions.

• W4: Since the key aspect of the viewability property is to enable high-quality reconstruction of $x$ from $\Phi(x)$ (L247, Sec 3.2), the viewability is quantitatively assessed by the reconstruction metrics, particularly LPIPS and FID. VisCoIN generally significantly outperforms the baselines thanks to the powerful pretrained generator. We'll make it explicit in Sec 4.1 to indicate "Fidelity of Reconstruction" quantifies viewability.

  We report below sparsity results of VisCoIN, FLINT for CUB at different relevance thresholds. The sparsity is calculated as the average number of relevant concepts per class such that global relevance $r_{k, c} > \text{threshold}$

  Method	threshold = 0.7	threshold = 0.5	threshold = 0.2
  VisCoIN	2.3	6.1	27.1
  FLINT	1.5	3.4	10.2

  FLINT achieves better sparsity because of its use of entropy based losses to compress $\Phi(x)$. While sparsity of VisCoIN could be increased by increasing the l1 regularization weight, we prioritized optimizing for reconstruction/viewability because (i) For previous unsupervised CoINs this is a major limitation, (ii) The current levels of sparsity seemed reasonable (total number of concepts K = 256 is much higher than relevant for any class), (iii) Excessive compression of information can make concepts less interpretable and similar to class logits. For completeness we will add this discussion in appendix.

Thank you for the review. We respond to your questions pointwise below:

Questions

• Q1: In our system design, we kept architectures taking $\Phi(x)$ as input as simple as possible to maximally preserve interpretability, i.e. $\Omega$ is a single linear layer and $\Theta$ is also a single linear layer with softmax. Although, we agree that even with a linear layer, interpretability for prediction can erode if the size of concept dictionary is too large and activations are not sparse. This is a limitation in general for all CoINs (supervised & unsupervised).

  • (1) We experimented applying a "Top-N" function on $\Phi(x)$ before $\Theta$, to keep only the most activated concept for prediction, for different values of N, and report results below. As mentioned by the reviewer, although it improves interpretability and conciseness of interesting concepts, it comes at the cost of accuracy of the overall system. However, we can see that using about 25% of the most activated concepts still preserves good accuracy in general. Dataset | N=4 | N=8 | N=16 | N=32 | N=64 | N=128 | N=256 --- | --- | --- | --- | --- | --- | --- | --- CUB | 23.75 | 42.25 | 59.28 | 70.07 | 76.25 | 78.97 | 79.44 Stanford Cars | 13.38 | 26.43 | 45.75 | 62.76 | 72.43 | 77.20 | 79.89 CelebA-HQ | 79.92 | 80.63 | 84.00 | 86.90 | 87.71 | x | x

  • (2) We include in Appendix F.6, an experiment where we directly use $\Phi(x)$ as latent vector $w_x$ for G, eliminating $\Omega$. While our model allows this design, it comes with certain limitations: (i) The user can't control the number of concepts. They are forced to employ a concept dictionary of same size as dimension of the latent space. (ii) Since the generator is pretrained and fixed, the resulting $\Phi(x)$ learnt is not sparse. (iii) Finally, in particular for GANs, it forcibly associates concept functions with columns of identity matrix as directions in latent space. Using an $\Omega$ (for instance a linear layer) allows the model to learn general directions in the latent space to associate to each concept function, which aligns with the conventional strategy for latent traversal inside GAN.

• Q2:

  • We are happy to report an additional experiment using a ViT-B/16 for $f$ on CUB dataset, while keeping other architectures almost identical. We started from a pretrained ViT-B/16 and only finetuned the classification head on CUB, to use as $f$. We take the patch embeddings of final layer as input to $\Psi$. We keep identical $\Theta, \Omega$, hyperparameters, and slightly modify $\Psi$ for reduced number of feature maps. As can be seen from the numerical results below, we achieve better accuracy thanks to a better pretrained $f$, but reconstruction and faithfulness are worse. The results could be improved by better desigining $\Psi$ and accessing more internals embeddings. However, they certainly show that VisCoIN can generalize to other backbone architectures. Model | Acc. f | Acc. g | LPIPS ($\downarrow$) | FF ($\tau = 0.2$) ($\uparrow$) --- | --- | --- | --- | --- VisCoIN - ResNet50 | 80.56 | 79.44 | 0.545 | 0.146 VisCoIN - ViT-B/16 | 86.66 | 85.86 | 0.582 | 0.081

  • Computational overhead of VisCoIN vs other unsupervised CoINs:

    • Number of trainable parameters: The subnetworks $\Psi, \Theta, \Omega$ are very light compared to $f, G$. The number of parameters is thus comparable or fewer than other unsupervised CoINs since both $f$ and $G$ are pretrained and fixed.
    • Training VisCoIN: VisCoIN is 2-4 times slower to train than other unsupervised CoINs because of the passes through the generator. However, we are still able to train VisCoIN on each task in a single day on a single GPU thanks to the use of pretrained $f, G$.
    • Inference, Interpretation time: The inference time is the same as other CoINs as it only uses $f, \Psi, \Theta$ to compute $g(.)$, and interpretation is similar or faster than unsupervised CoINs, since no input optimization is required.

• Q3: We manually added red boxes to indicate the main regions where the modification is occurring in generated images. We will improve the figure caption and hope that it is more clear.

Thanks for the review. We address the concerns pointwise below:

Weaknesses

• W1: Compared to other unsupervised CoIN, we make significant advancements in reconstruction, as assessed by the different reconstruction metrics, but we agree it is not perfect. While concept attribution maps are very useful to highlight regions of relevance for a concept, similar to feature attribution approaches, they are not as effective in revealing the "semantic content" detected by a concept. Our intervention and visualization approach for interpretability is aimed at filling this gap. Our approach can be useful even if the reconstruction is not perfect, provided it's close enough so that changes in it can be grounded to the original image. Most importantly though, we do not believe these two are in competition with each other but complementary. We raised the point about using a tool to highlight relevant regions for a concept in Appendix G.2 such as the difference of images to assist in localizing the modifications.

• W2: The unconstrained supporting representation leverages specificities of the StyleGAN family of architectures. While it improves reconstruction quality and results, it is not crucial to the understanding and proper learning of the overall system, and we thought it would hinder clarity in the main text. We show in Appendix D that the system provides meaningful results with experiments using ProgressiveGAN and $\beta$-VAE that do not rely on $\Phi^\prime$. Nonetheless, we will add a reference in Section 3.2 to the detailed discussion in the Appendix.

Questions

• Q1: We think that the constraint on $\lambda$ comes from the limitation of doing linear traversal in the latent space of G. An unreasonably high $\lambda$ can push the resulting latent vector outside the image manifold relevant to the dataset. Setting $\phi_k(x)=0$ generally should not cause problems as it corresponds to moving towards average latent vector of the generator, which is generally the "centre" of the latent space. Also, since only a fraction of concept dictionary is generally relevant for any class (more details in our response W5 to Reviewer Lumc), any given $\phi_k(x)$ is frequently 0 for many input samples.

• Q2: We only considered designing $\Omega$ with single fully connected layers for experiments, such that it associates each concept function $\phi_k$ to a linear direction in latent space of $G$. Intervening on concept activation $\phi_k(x)$ corresponds to linear traversal in latent space, which is a common practice for latent traversal in generative models. An arbitrary complex transformation would result in reduced interpretability of image transformations.

  However, it is an intriguing suggestion, as recent work has shown that linear trajectories are not always optimal for latent traversal, which. We discuss this as a limitation in Appendix H. More complex design of $\Omega$ could achieve meaningful non-linear traversal and improve reconstruction. We leave it as a future work since its a challenging problem in itself.

Thank you for the review. We address your comments below:

Weaknesses

• W1: Yes, one can consider using cross-entropy loss with ground truth labels instead of "output fidelity loss". We specify the possibility to use both when describing the general architecture of unsupervised CoINs (Line 196). We decided to use the output fidelity loss following previous work (Sarkar et al., 2022). The current design also draws inspiration from knowledge distillation setting, in which a student model is trained to reproduce output of a teacher model. One additional perk of using this "output fidelity loss" during VisCoIN training is that it can also be applied for images without annotations, such as images sampled from G (further details about VisCoIN training in Appendix C.2.3). This provides additional guidance and stability to train $g$. For completeness, we include here an experiment of VisCoIN trained using a standard cross-entropy loss with ground truth labels on CUB dataset: Model | Acc. ($\uparrow$) | LPIPS ($\downarrow$) | FF ($\tau = 0.2$) ($\uparrow$) --- | --- | --- | --- VisCoIN - Output fidelity loss | 79.44 | 0.545 | 0.146 VisCoIN - Cross-entropy loss | 78.89 | 0.559 | 0.076

• W2:

  • We discuss in the limitations (Appendix H) the inherent limitations for disentangling concepts in the unsupervised learning. However, it is also a problem appearing in supervised learning of the concepts (with supervised CoINs, e.g., CBMs) that can suffer from concept leakage [A]. Furthermore, increasing the size of the concept dictionary (i.e. $K$, the number of concepts in $\Phi$) could help in learning more disentangled concepts at the cost of reducing conciseness of the dictionary, similarly to Sparse AutoEncoders (SAEs) in "mechanistic interpretability" [B]
  • We are happy to report an additional first experiment with a transformer architecture. We started from a pretrained ViT-B/16 and only finetuned the classification head on CUB, to use as $f$. We take the patch embeddings of final layer as input to $\Psi$. We keep identical $\Theta, \Omega$, hyperparameters, and slightly modify $\Psi$ for reduced number of feature maps. As can be seen from the numerical results below, we achieve better accuracy thanks to a better pretrained $f$, but reconstruction and faithfulness are worse. The results could be improved by better desigining $\Psi$ and accessing more internals embeddings. However, the results certainly show that the idea can generalize to other backbone architectures. We discuss desiderata for $G$ and using diffusion models, in Appendix B. They would be interesting to explore as an extension, but despite recent positive steps towards understanding their latent space and discovering meaningful latent directions ([C], [D]), it is currently difficult to design an $\Omega$ that allows convenient latent traversal. Model | Acc. f | Acc. g | LPIPS ($\downarrow$) | FF ($\tau = 0.2$) ($\uparrow$) --- | --- | --- | --- | --- VisCoIN - ResNet50 | 80.56 | 79.44 | 0.545 | 0.146 VisCoIN - ViT-B/16 | 86.66 | 85.86 | 0.582 | 0.081

[A] M. Havasi, S. Parbhoo, F. Doshi-Velez. "Addressing Leakage in Concept Bottleneck Models". NeurIPS 2022

[B] E. Nelson, et al. "Toy models of superposition." arXiv preprint arXiv:2209.10652 (2022)

[C] Y.H. Park et al. "Unsupervised Discovery of Semantic Latent Directions in Diffusion Models". NeurIPS 2023

[D] M. Kwon et al. "Diffusion Models Already have a Semantic Latent Space". ICLR 2023

Weaknesses

We thank the reviewer for their insightful comments. We answer pointwise below, and will add the discussions in the revised version of the paper.

• W1: In relation to LLM/VLM based concept bottleneck models (CBMs), we present below arguments why visualizing concepts is still important:

  • When considering expert or domain specific datasets like Stanford Cars (for car models classification) or the MVTec Anomaly Detection [A] (for anomaly detection of object in production lines) for instance, visualizing concepts directly on the objects is simpler (and faster) to understand for human operators, rather than reading a text description.

  • For certain computer vision applications (eg. self-driving cars, medical imaging tasks), visualization provides spatially localized interpretations, which is more difficult and cumbersome with text. For instance, if a concept relating to "red light" is activated for an image, to get a thorough understanding of the model's decision, it is crucial to identify which regions and what content in those regions activates the concept.

  • The LLMs/VLMs which the recent CBMs are based on (particularly CLIP) are limited when detecting concepts and image details at a finer spatial scale [B].

  • As discussed in Appendix A, the current methods are prone to generating concept descriptions not grounded in any visual information, which also harms their interpretability. Take the following example. We have an image of a dog, and want to know which concepts lead to the classification of the image as a dog. An LLM or VLM might, for instance, introduce concepts such as "Loyal/Honest" for detection and interpretation, ie: "the object in this image is loyal, therefore it is a dog". In this situation, "loyal" should not be even generated as a concept, since it can never be visually observed.

  • In the case of LLM/VLM based CBMs, there are also concerns about faithfulness of concept detection to the text description. This is a similar issue to concept leakage [C]. We believe that the ideas presented in our work, such as viewability, can help in identifying such issues in LLM/VLM based CBMs.

  [A] P. Bergmann et al. (2021). "The MVTec anomaly detection dataset: a comprehensive real-world dataset for unsupervised anomaly detection."" IJCV.

  [B] C. Gou et al. "How Well Can Vision Language Models See Image Details?". https://arxiv.org/pdf/2408.03940

  [C] M. Havasi et al. "Addressing Leakage in Concept Bottleneck Models". NeurIPS 2022.

• W2: The methods mentioned, among others, are separately defined by us as "supervised CoINs" (Section 3.1), since they use concept annotations to learn $\Phi$. Although these concepts annotations can be "automatically" obtained from LLMs and VLMs, they are extracted/generated beforehand to train the underlying "CBM". On the other hand, all the baseline methods we compare to "discover" concepts in an unsupervised way without concepts annotations, which we define as "unsupervised CoINs".

  A major difference which makes adapting or comparison with them difficult, is their inherent lack of "decoder" model, that prevents us from going back to the input space from the concept activations. Without a decoder, we cannot evaluate consistency, faithfulness, and reconstruction metrics, making a comparison only available for accuracy. We will add this clarification in the paper.

• W3: Thanks for the remark. We'll improve the clarity by better separating the sections, discussing experiments in more depth, and add a summary of take-aways for them.

• W4: Thank you for the reference. It is indeed an unsupervised CoIN that includes a GAN as a decoder model. However, it differs significantly from VisCoIN in two major aspects. First, they learn the generative model simultaneously along with other constraints. This can make the overall training challenging, since GANs are notoriously difficult to train. The training will also be more costly for large scale images and bigger generators. Second, they do not leverage the GAN for visualizing the concepts. They only use the maximum activating samples (MAS) for visualization. Thus viewability is not part of their aims. The GAN is used as a decoder with higher expressivity, with the goal of improving accuracy.

  Unfortunately, we didn't find any public codebase to help reproduce and compare with this method. The experiments in the paper are only on small scale image datasets like CIFAR10 and CIFAR100.

Questions

• Q1: In our experiments, we finetuned both $f$ and $G$ on the datasets being studied, before training VisCoIN. We didn't study how much these restrictions can be lifted. We expect that the pretrained $f$ and $G$ should have good accuracy and generative capabilities on the dataset being studied even if they aren't trained on it. However it's an interesting future research direction to explore. Thanks for raising the point.

I thank the authors for the responses, and have increased the score to 6.

While I understand the manuscript cannot be updated any further, my point about W2 was that -- if you adapted (eg. attached a simple decoder to) an existing method such as Label-free CBMs, would that outperform such an approach? I could not follow why this is non-trivial. I'd appreciate if a discussion on this can be added, but I'd understand otherwise too.