Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?

Thank you for the feedback! You will find our point-by-point responses below.

I could not find the precise training/validation procedure used for the baseline "CBM val".

We have updated the public repository to include the configuration file config_synthetic_cbm_val.yaml and code to run the “CBM val” baseline. As mentioned, CBMs can be viewed as classification heads on top of pre-trained models. However, this would correspond to the “post hoc CBM”. We frame the presented CBM in its original “ante hoc” implementation [6], where the backbone is trained from scratch. Our work focuses on making interventions on any pre-trained black box more effective, not the CBMs. From the options provided by the reviewer, our “CBM val” is number 4. This way, we show the sample efficiency of our method over the (ante hoc) CBMs when only a small portion of annotated data is available.

“Since interventions can be executed online using a GPU, our approach is computationally feasible”: This is not a formal justification for why the proposed approach is computationally tractable.

This statement is not meant as a formal complexity metric, but as a remark for the interested reader that the interventions are less costly than training or fine-tuning and can be run “online” with a regular GPU. As the intervention procedure is gradient-based, the computational cost varies across datasets and depends on the optimization iterations. The number required depends on several factors, e.g., the value of $\lambda$, the number of concepts, the dimensionality of $z$, and the number of data points per batch. Given these, providing a formal description of the running-time complexity is challenging. For example, on the synthetic dataset, with the convergence parameter set to $10^{-6}$ (very conservative) and for a batch of 512 samples, the intervention procedure requires approx. 500 to 2500 steps (the number varies depending on the number of given concepts and the value of $\lambda$), which amounts to 0.5 to 2 s (for the whole batch) for an untuned black-box model. We use smaller batches and more permissive convergence criteria when fine-tuning, allowing a considerable speedup. In addition, note that the run-time of the intervention procedure is not strictly dependent on the black-box model’s architecture (except for the dimensionality of the layer intervened on).

“This work focuses specifically”: This paragraph is already quite technical. While clear, it could be moved to a "preliminary" section after the introduction.

Thank you for the suggestion! In order to address a broader audience, we will include a preliminary section introducing the method with the technical contributions, and reframe the introduction to be less technically oriented, removing the formal notation.

Figure 2, The arrow in step 3 might go downwards.

We will update the figure.

“For a trained CBM fθ (x) = gψ (hφ (x)) = gψ (ˆ c)”: Consider explaining again here what c^\hat{c}c^ represents

With the new organization of the introduction and preliminary section of the methods, we will introduce the notation accordingly prior to the description of the method.

“we define the intervenability as follows”: Consider explaining the idea behind the equations before showing the equation to ease reading.

Intuitively, the more effective an intervention is, ideally, the more it should reduce the target prediction error. Leveraging this, our equation measures the gap between the prediction error prior and post intervention, where larger gaps correspond to more effective interventions.

I. “where D is the joint distribution over the covariates …”: Consider describing the variables involved in an equation before showing the equation to avoid the reader going back and forth. II. “To alleviate the need for human-annotated concept values”: This sentence might be confusing as it is not clear, I'd rephrase it. III. Table 1: Consider dropping "0" at the beginning of each value or express numbers in base 100 to save space.

We will clarify and introduce the notation in I, rephrase II, and change the metrics to percentages and bold the best results in III.

CIFAR 10 results in Table 1: Why not use CIFAR-100 (where concept labels are available as a ground truth) instead of CIFAR-10?

We include experiments with both CIFAR-10 and ImageNet that do not originally have available ground-truth concepts to show that our method can be combined with “concept discovery” [7], where annotations are acquired through VLMs. We did not include CIFAR-100 as we have a synthetic tabular, two natural image datasets (CUB and AwA2), and two medical image datasets (MIMIC-CXR and CheXpert) that, we believe, are sufficiently representative to cover the settings with “annotated” concepts.

The largest gap with respect to baselines is on AwA and MIMIC. Do you have an intuition why this is the case?

Generally, our procedure has a stronger effect because we target the improvement with interventions with a stronger inductive bias, while other methods do not explicitly encourage intervention effectiveness. In some instances, activations are correlated with concepts, but the network does not rely on them for prediction, e.g., when there is a complex relation between the concepts and target, as in the case of X-rays (MIMIC), and only probing is not enough to intervene effectively. However, after fine-tuning, the black box learns to use the information correlated with the concepts at prediction time. At the same time, our approach does not require introducing a concept bottleneck layer, and, therefore, the model may still use additional non-concept-based information from the representations. With respect to the AwA (and CIFAR-10), the experimental findings fall within our expectations: in both, concepts are useful for predicting the target variable, and our fine-tuning explicitly encourages intervention effectiveness, providing a stronger supervisory signal.

Thank you for your nice feedback and for your interest in our work!

As a follow up to the two raised points:

• Computational cost: We agree with your point and will include the discussion in the appendix, thank you! Regarding the most critical factors, we believe all, the convergence parameter, number of concepts, number of datapoints, value of lambda, and the dimensionality of z play a relevant role. Due to the nature of the gradient-based optimization that runs until convergence, it is challenging to provide a formal estimation of the cost or ranking. However, one could try to approach the problem empirically by providing wall-time results of the optimization under controlled synthetic experiments varying the mentioned parameters to provide a better understanding of their relevance, which we will include in the appendix of the revised paper. We will, in addition, consider characterising more formally the complexity of a single update step in this gradient-based procedure, which should give additional intuition on the complexity.

• CIFAR: We believe that the original concepts coming from CIFAR-100, i.e. the classes, and those from [7] in the VLM scenario differ significantly. More precisely, the original work in [7] introduces a total of 824 concepts in CIFAR-100 while the original annotated labels are only 100, and cover different categories. We assume in this discussion that the “concepts” the reviewer refers to in CIFAR-100 are the super-classes. For this reason, using the VLM-based concepts to intervene on a model trained with the original CIFAR-100 concepts would not be possible. However, we envision a setup where keeping the original concept categories, the CIFAR-100 dataset can be re-annotated using CLIP, this way the experiment proposed by the reviewer could be conducted. Based on our experience with VLM annotated data, we would expect it to have more noise and, therefore, potentially, less effective interventions than under the ground-truth annotations.

Thank you for your detailed comments! Below, we respond to your concerns point by point.

Equation 4: presenting Eq. 4 and then only considering the special case of β=1 is not correct.

We believe the formalization of the overall optimization problem is a beneficial contribution to future lines of work. However, we agree our focus is on the simplified case and will adapt the manuscript first introducing the special case, followed by the general formulation.

In the abstract, “Recently, interpretable machine learning has re-explored CBMs”. The sentence is not very clear, it seems in this way that CBMs have already been proposed in the past.

We refer to the works referenced in “Introduction” and “Related Work” [4,5] and will make them more explicit. They propose concept-based approaches to classification. Although not strictly referred to as CBMs, they are very similar and the original work on CBMs [6] acknowledges them as closely related.

“While a specialist may just override predictions, concept and target variables are linked via nonlinear relationships potentially unknown to the user.” This sentence is not super clear and does not seem to support the paper point.

We motivated the need for concept-based interventions, especially in scenarios where a professional has the concept information but inferring the target variable is not an easy task. We understand the potential confusion and will remove it.

Figure 1: The concepts reported are not clear.

We will include their description in the main text and caption. Particularly, in order of appearance from left to right, and top to bottom, the meaning of the icons is: “fierce”, “timid”, “muscle”, “walks”, “otter” and “grizzly bear”. This figure is complemented by a more concrete example of model correction in Figure A.2 with a list of all intervened on concepts.

“think of a doctor trying to interact with a chest X-ray classifier by annotating their findings”, which findings?

The findings are the concepts of the X-ray datasets. We will provide examples to help understand the X-ray use-case and include a table in the appendix with the full list of concepts: atelectasis, cardiomegaly, consolidation, edema, enlarged cardiomediastinum, fracture, lung lesion, lung opacity, pleural effusion, pleural other, pneumonia, pneumothorax, and support devices.

Not clear whether you consider both a linear and a non-linear probe, or only one of the two without looking at the experiments.

We will clarify that the experiments shown in the main body are performed under a linear probe and an ablation with a nonlinear probe is found in the appendix.

You should probably cite also TCAV in 3.1 when talking about probing. The CAV is probably the most renowned concept probe.

We will include it.

I think you should better explain the fact that if you directly optimize equation 3 you could decrease the performance of the model.

We will make it more explicit that the final loss function is a linear combination of the intervenability measure in Eq. 3 and the target prediction loss, resulting in Eq. 4. We always consider the combination of the two to better control the tradeoff.

Fine-tuned, A: explanation to why including this model

The motivation is: it would be natural to include the concepts in the model, either at the input or at intermediate layers. Since the considered feature spaces are high-dimensional, we append concept variables to the intermediate activations as a direct comparison to Fine-tuned I.

“Finally, post hoc CBMs exhibit this behaviour: interventions have no or even an adverse effect on performance.” This claim is not supported by the results presented in figure 4(d) bottom.

We will rephrase the statement to clarify that in some datasets, a small positive effect is present.

Table 1 report results where the black-box model is normally not that good for the target performance.

In synthetic data, the generating process favors the CBM architecture, as concepts directly relate the inputs to the target and we expect the CBM to outperform. In the remaining datasets with more complex concept-to-target dependencies, the CBM performs worse, as expected. If the concept variables are “useful” for the target prediction, we hypothesise the fine-tuning methods would improve predictive performance.

Specifically, fine-tuning enforces reliance on the concept variables, which might increase the robustness and prevent overfitting. Moreover, fine-tuned black boxes keep the original representation without introducing a bottleneck layer and can still use non-concept-based information if necessary. To finalize, our focus was on making the models more intervenable, and Table 1 suggests that most methods behave on par, i.e., fine-tuning with intervenability does not harm the performance.

Why is the CBM more computationally expensive in the experiments with VLM-based Concept? I think you could use a fixed CLIP to predict the concepts.

The VLM-based concept annotation does not influence the computational cost of the CBMs. We have used these annotations to explore larger backbone architectures and training these from scratch, as would be needed for the (ante hoc) CBM baseline, is not computationally feasible. It would require a large number of concept annotations and resources. Using a fixed CLIP to predict the concepts is alike the post hoc CBM, which we include as baseline.

In Cher X-ray classification the black-box models are not intervenable. This partly contrast with the claim that you propose a method to make “interventions on any pretrained neural network post hoc.

In some instances, activations are correlated with concepts, but the network does not necessarily rely on them for prediction, hence, only probing is not enough to intervene effectively. However, after fine-tuning, the black box learns to use the information correlated with the concepts at prediction time. Hereafter, fine-tuning may be necessary in some instances.

We thank the reviewer for the feedback and questions! Below is our point-by-point response.

There are two main advantages of using CBMs, interpretability and intervenability. While the proposed method allows for intervenability, this method does not add interpretability to black box model.

As stated by the reviewer, this work focuses on intervenability and its formalisation. However, the proposed method also incorporates interpretability with respect to a regular black box model. By mapping intermediate layers to human-understandable concepts via the probe, the user gains a better interpretation of the information contained in the representations. This connection between probing classifiers and interpretability is supported and further detailed in prior work referenced in the manuscript [1,2,3].

One could argue that since you need a validation set with concept anyhow, one should just put a CBM in the original architecture giving both interpretability and intervenability. Empirical validation in the paper did show that the proposed method did better than CBM however baseline CBM used in the evaluation was not trained for intervenability as done by Mateo, et al. [1] which could dramatically improve the performance of regular CBMs. So its difficult to conclude that the proposed method is in fact superior to CBMs in any way.

The original CBM architecture requires training the concept and target predictors from scratch. This is a limitation in the settings where there is not enough data available to train a large backbone, e.g. using Stable Diffusion like in the CIFAR-10 and ImageNet experiments. This problem would be aggravated further if only a validation set with concept annotations was used. We explore this in Figure 3 and show that CBMs are not sample efficient when trained only on the validation set.

We did a preliminary exploration of the introduction of interventions at training time in the CBM, similar to the work done on the mentioned CEMs by Mateo et al., and we did not find a significant improvement with respect to the regular CBM in the presented datasets.

Effectively, intervening on a vanilla CBM at training time can be thought of as a combination of independent and joint training, which, as shown in Appendix F.10, have similar results.

In the cases where the concept set is incomplete, the bottleneck layer in the CBM will hinder the overall model performance, even under interventions. However, finetuning for intervenability does not limit the expressivity of the intervened-on layer and, therefore, allows for better target prediction.

Which layer of is used for interventions, does it matter?

The reported experiments show the results of intervening two non-linear layers after the backbone. The layer you intervene on will play a role in the effectiveness of interventions. Generally, high-level information in neural networks emerges in deeper layers, which is why we recommend conducting the interventions there. Additionally, the required complexity of the probe is lower as the dimension of the layers is reduced deeper within the network. Prior works show probing at deeper layers is benefitial [2]. However, the proposed method can be applied at any step, provided the layer has extracted enough information, and we expect interventions to have a positive effect on the target prediction.

For the multi-task baseline, Fine-Tune MT how do you intervene on test time since changing the multitask label does not affect the output of the classifier?

The intervention on Fine-Tune MT is done in the same fashion as in the proposed Fine-Tune I, using the three-step solution introduced in Section 3.1. in the manuscript. This will be made clarer in the revised version of the paper.

For the appending baseline Fine-Tune A do you throw away the last layer of the original network (i.e the layer after the one used for interventions) and create a new one to handle the new size?

This is correct. We leverage the pretrained backbone until the layer where the concatenation with the concepts occurs, in this case, two non-linear layers after the extracted features. At this state, a new target predictor is created with input layer size correspondig to the representations concatenated with the concepts.