DCBM: Data-Efficient Visual Concept Bottleneck Models

Thank you for your detailed feedback and for recognizing the flexible and interpretable design of DCBM. We highly appreciate your helpful comments and hope to provide the missing details in our answers below.

Efficiency

We derive concept proposals based on 50 samples per class and train the CBM on all training samples. However, as we show in response to Review 3 (AELC), DCBM's performance does not deteriorate when we train with only 50 images per class. In contrast to BotCL (Wang et al., 2023), we do not fix the number of concepts, but the number of images that we use for concept extraction. This design choice avoids BotCL's weakness (Wang et al., 2023) of having to tune the number of concepts for each dataset.

When comparing DCBM to BotCL, we quantitatively outperform them on CUB (8.4%). Our results for ImageNet are not comparable, as they evaluate only 200 out of 1000 classes and report that their method fails for a large number of classes. Given that BotCL's authors do not share their training recipe, we cannot evaluate overall efficiency.

Question: Would you like us to report an evaluation of DCBM on the first 200 classes of ImageNet?

Ablation on the number of concept samples

Table 1: Ablation for DCBM w/ GDINO Partimagenet on CUB (CLIP ViT-L/14)

In section D.5 we ablate the number of concept samples, which have - in parts - copied here for your convenience. We will add a reference to Appendix D.5 in the paper's section 3.2, where we describe the concept generation process.

Concept validity

We validate our concepts empirically, by calculating the energy pointing game (GridPG) (Bohle et al., 2021). Our motivation for this choice is in line with BotCL (Wang et al., 2023), we want to verify whether detected concepts can be traced back to the image region. In DCBM, the concept-image alignment is verified as part of the evaluation. We chose an automated evaluation over a human analysis.

Theoretical explanation of segmentation

Our understanding is the following: An object $O$ can be represented as a combination of concepts from a global concept pool $\mathcal{C}$. Let $C_i \in \mathcal{C}$ denote individual concepts, and let each object be characterized by a subset of these concepts with corresponding weights.

where:

• $C_i \in \mathcal{C}$ represents a concept from the global pool,
• $w_i$ denotes the contribution of concept $C_i$ to the image,
• $C_i$ is located in the image $I$, $R_i \subseteq I$ is the region of the image where concept $C_i$ is present,

Further, we assume visual concepts to be spatially localized in images and therefore conjecture that data efficient concept extraction should build upon image segments or regions rather than entire images. Therefore, we generate the global concept pool $\mathcal{C}$ by segmentation or detection foundation models, which are then cropped out and used as concept proposals $s_i$. All $s_i \in \mathcal{S}$ are then clustered into the global concept set $\mathcal{C}$.

In A. Algorithms (appendix) we provide a further theoretical explanation of this process along with pseudocode. We will update this section following your feedback. Would you like us to include any additional details? Thank you for carefully reviewing our methods section - we have revised the paper and unified functions and symbols.

Additional datasets

DCBMs stand out by applying to any domain in a data-efficient manner. We show on 7 diverse datasets, that DCBMs can be applied to animals (CUB, ImageNet), scenes (Places365), social media (ClimateTV), low-resolution images (cifar10 & cifar100) along with ood generalization (ImageNet-R) and state changes (MiT-States). Additionally, we show for 2 novel datasets in the rebuttal, that DCBMs performance is independent of the domain it is trained on. This exceeds the number of datasets BotCL evaluates, i.e. 4. Our main evaluation contains the same datasets as employed for Vlg-CBM (Srivastava et al., 2024) and the same number of datasets as Res-CBM, i.e. 7 datasets. We include these models in our related work and compare against them where possible.

We thank you for suggesting additional experiments on AwA2 and CelebA. We have run DCBM on AwA using the standard 50:50 train and test split. We created the val set by randomly selecting 10% of train samples.

Table 1: DCBM performance on AwA2 using GDINO (w/ partimagenet labels) as concept proposal method.

For CelebA, the experiments are currently running.

Misc

Thank you for your interest in seeing more concept visualizations. We have included more examples in the supplemental material.

Thank you for reviewing our work. We appreciate your positive feedback on our concept extraction pipeline that leverages foundation models, and we’re pleased that you recognize the interpretability and localization capabilities of our DCBM.

Data efficiency in real-world setting

DCBM is designed for real-world data scarcity, relying on only 50 samples per class during the concept proposal generation phase. Notably, our experiments also show robust performance with even fewer samples in Appendix D.7.

This can be taken one step further by reducing the number of training samples to the same subset. For ImageNet, we reduce the number of training samples from originally 1,281,167 images to 50,000 images (-96%), and the performance degrades from 77.4% to 75.0%. For Cifar10, the reduction in training images from 50000 to 500 (-99%) reduces the performance from 97.5 to 93.1. The experiments were run using GDINO (partimagenet) and CLIP ViT-L/14. We will include this experiment for all datasets in the paper, thank you for the suggestion.

The focus of our work is data efficiency with comparable performance levels. We provide the results of another, non-overlapping 50 images subset (RQ2) and report the accuracy when training the CBM with the segments of 100 images per class, combining the two subsets (RQ1). For CUB, the training set consists of less than 50 images per class, thus we already include all images in the concept proposal generation.

Table 1: Performance evaluation of subset selection

In Table 1, s1 corresponds to the original set of 50 images per class, s2 represents an additional set of 50 randomly selected images per class, and s1+s2 denotes the combined dataset of 100 images per class.

The performance is stable between all subsets. We believe that the performance difference to DN-CBM is domain-dependent. Given their vast number of pre-training images (3.3M), general domains are well covered (ImageNet/Places365) whereas specialized domains benefit from the dataset-specific DCBM (CUB).

Comparison to DN-CBM

We agree that the CBM training for DN-CBM and DCBM are quite similar. However, the required steps prior to the CBM training differ significantly. While DN-CBM trains a SAE to retrieve concepts using 3.3M images in CC3m, we create the segments by applying foundation models to a subset of the training images. We would like to highlight, that for DN-CBM, the download of an additional 3.3M images is needed, whereas, for DCBM, we only require the dataset to be analyzed. This reduces the storage requirements significantly and is highly time efficient.

Table 2: CBM training preparation: DN-CBM vs DCBM

Generalization capabilities

We agree with your criticism of our reporting of DCBM's generalization capabilities. We would like to point you to Table 19 in the supplementary material where we can report a 22-27% error rate difference between ImageNet and ImageNet-R for CLIP-ViT/L14. For this embedding model, we achieve error rates of below 50% when evaluating on ImageNet-R. Due to resource sparsity, only the results for CLIP-RN50 were ready at the time of submission. We are currently generating the accuracy for DN-CBM and will provide them asap.

Misc

Thank you as well for pointing out that some typos and ambiguities exist in the paper - we have fixed them. We believe that your feedback has helped us to improve our paper and would like to thank you for taking the time and sharing your expertise.

Thank you for your thoughtful and constructive feedback. We appreciate your recognition of our framework’s originality and data efficiency—requiring 10x fewer concept labels than comparable CBM approaches while achieving similar performance. Below, we provide detailed responses organized by the key points raised:

User Studies and Quantitative Metrics for Domain Utility

We agree that demonstrating practical utility for domain experts is crucial. While our primary focus has been on establishing the theoretical feasibility and data efficiency of our approach, we recognize the value of user studies and quantitative metrics for concept intervention. In the final version, we will include an additional investigation that actively manipulates relevant concepts to observe changes in the model’s predictions and confidence. For instance, as shown in the bird example (concept “rock”), our preliminary analysis indicates that concept removal has a predictable effect on predictions. We plan to extend this analysis by adding concept interventions to further validate interpretability. In addition, we recognize that the risk of "explanation illusions" is an important concern in CBM approaches, particularly when using CLIP as a backbone. To mitigate this, we plan to incorporate tests of concept removal and alteration to explicitly address concept grounding.

Analysis of the Sparsity Mechanism and Concept Coverage

Analog to concurrent work, we ablate the sparsity parameter $\lambda$ based on model accuracy (Rao et al., 2024 & Oikarinen et al., 2023)

Concept-Level Validation and Baseline Comparisons

We acknowledge that thorough concept-level validation is essential to reinforce the interpretability of our method. Our study has systematically analyzed the learned concepts across multiple large-scale datasets (ImageNet, Places, and CUB).

We deliberately compare only methods that offer interpretability, which is a core property of CBMs. By focusing on interpretable approaches, we ensure that our evaluation remains consistent and that our performance improvements are attributable to our architectural innovations rather than differences in method transparency. This is why we do not evaluate against prompt-tuned CLIP. We will include this distinction in the literature review.

However, as demonstrated in [fCuP], we incorporate additional datasets and utilize new, randomly selected images for the concept proposal generation phase. Our results consistently show that our technique delivers comparable performance across diverse experimental settings.

Broader Applicability Beyond Classification

Regarding the current focus on classification tasks, we appreciate the suggestion to explore applications in regression and other settings. However, the investigation of the suitability of CBMs in general is an additional task, which takes more consideration than would be adequate for the purpose of this paper. Therefore, we will include this discussion at the end of our paper to open up new research fields of CBMS for regression and other settings.

Thank you again for your valuable feedback. We believe these planned additions and clarifications will further strengthen the work, and we look forward to incorporating your suggestions to improve both the interpretability and practical utility of our approach.

Thank you for your detailed and constructive feedback. We appreciate your acknowledgment of DCBM's strengths being simple, adaptable, and avoiding large-scale pre-training. Below, we outline our responses structured by the key points you raised:

Performance on Large-Scale Datasets

In our opinion, while CLIP is powerful and designed to capture relationships between images and text, projecting concept centroids into this space might not perfectly align with the underlying semantics of the image data. Additionally, large-scale datasets like ImageNet and Places365 contain a high degree of intra-class variability, making it difficult for a concept-based model to generalize effectively. One could improve the discrepancy within large-scale datasets by adapting the centroids gradient-wise during the training phase of the linear layer. Here, the centroids would dynamically adapt to the underlying structure of the dataset.

Segmentation Challenges in Specialized Domains

It is one strength of DCBM that the segmentation model can be exchanged by a domain-specific one, e.g. MedSAM (Ma et al., 2024) for the medical field—offer. We believe that incorporating such domain-adapted segmentation models can help address challenges related to identifying relevant regions in specialized tasks.

We agree that combining textual domain-specific concepts with segmentation-based visual concepts may be especially beneficial in the medical domain. Some concepts are more effectively expressed visually, while others are better captured through language. In DCBM, it was our primary objective was to develop a CBM that operates independently of textual inputs or LLMs for extracting concepts. We believe, that the combination of DCBM with existing methods like LaBo (Yang et al. 2023) and LF-CBM (Oikarinen et al. 2023), would achieve such a combination.

Extension to PCBMs and Backbone Flexibility

DCBM cannot be extended as a post-hoc CBM - as neither of the other ante-hoc CBMs. We have chosen this approach as it allows to better understand the model embedding space. This said we believe that DCBM is a valuable framework to better understand any vision embedding space. We utilize text-image aligned backbones (CLIP) for benchmarking against other CBM approaches. This is not inherent to our framework and DCBM can be run using any vision embedding, with a slightly more intricate mapping to the text labels.

We appreciate the recommendation of Moayeri et al. (ICML 2023) and agree that projecting embeddings from a blackbox into the VLM embedding space is a compelling strategy. We find this approach both intriguing and valuable and are investigating the opportunities of combining post-hoc and ante-hoc CBMs.

VLM Biases and Spurious Correlations

The CLIP space is known to contain spurious correlations (Rao et al., 2024 & Oikarinen et al., 2023 & Panousis et al., 2023). By including all segments as concept proposals, DCBMs visualize spurious correlations. As shown in Figure 3, DCBM learns that rock is an important concept for predicting the bird class. This further becomes apparent, when we set the weights of rock to zero, the model achieves a confidence of 62.49% (-12%). We are currently training a model with interventions.

Question: We exclude concepts from training, which have been identified as spurious. Alternatively, we were thinking of masking out concept regions in the image and the measure the model's confidence. Is this what you had in mind? We would love to hear more feedback on this from you.

In the final version of this paper, we further investigate the behavior of spurious correlations and actively remove such concepts to observe changes in accuracy.

Systematic Identification of Spurious Concepts

This is possible by using either slot attention as in BotCL (Wang et al., 2023) or an image tagging model such as RAM (Zhang et al., 2024) with a threshold. We will include this extension in the discussion section.

We appreciate your insightful feedback and hope our responses have effectively addressed your questions. During the remainder of the rebuttal phase, we welcome further discussion on any open issues and invite additional feedback on our proposed experiments.