Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?

We thank the reviewer for the thorough feedback! Below are our point-by-point responses to the concerns raised.

The work seems to have missed the most relevant work on conceptual (interventional) counterfactuals, e.g., [1,2]. While there are some technical differences (usage of the gradients stemming from the linear probe instead of the classifier in the counterfactual optimization problem), there is still significant overlap. For example, Abid et al. [1] also use linear probes to identify concept activation vectors and use it to intervene on the features.

We thank the reviewer for pointing us to this line of work! In the revised manuscript, we have included references to [1] and [2]. We have also provided a detailed discussion of the differences between ours and these related works in Appendix B. Below, we provide a general summary.

While some of the technical tricks used by our work and conceptual counterfactual explanations (CCE) [1, 2] are similar, the problem setting and the actual purpose are quite different. Informally, our intervention addresses the following question: “How do the network’s representations need to be perturbed to reflect the user-input concepts $\boldsymbol{c}’$?” On the other hand, CCEs try to tackle the following question: “Which concepts need to be input for perturbing the network’s representations to flip the predicted label $\hat{y}$ to the given $y’$?”. In summary, CCEs try to identify concept variables responsible for a misclassification, whereas interventions try to improve a prediction at test time based on the input concepts.

In our method, interventions are intended for the interaction between the user and the model so that the user can reduce the model’s error via understandable concept variables. By contrast, CCEs consider a typical counterfactual explanation scenario, trying to identify which concepts can lead to sparse and in-distribution changes in the representation that would flip the classification outcome.

Beyond the main difference above, our work makes a few other technically valuable contributions. Namely, we formalise the intervenability measure and intervention strategies and propose explicit fine-tuning for intervenability. In summary, our work tackles a different problem complementary to the direction explored by [1] or [2].

The work motivates their approach by stating that “concept labels are not required [during training]” (p. 2). While true, it is still required for the fine-tuning (as we need to train the probes initially; Sec. 3.3, and shown to be important in the experiments), which one could also see as part of model development. For the case that we would like to make the same conceptual interventions as for CBMs, this would result in a similar amount of annotation cost.

Fine-tuning indeed requires concept labels. We would like to emphasise that all the experiments were performed on validation sets, which were considerably smaller than the training sets (utilised by CBMs). Moreover, labelling costs are not the only concern when using CBMs. In particular, concept variables might be unknown when training a model and only be incorporated when deploying the model for a specific application in some institution (e.g. hospital). For instance, such challenges may arise when utilising foundation models [5].

The paper makes the (implicit) assumption that the feature extractor $h_{\phi}$ learns the concept; both in their intervention approach and fine-tuning (since $\beta$ is set to 1). However, the authors provide no evidence that this is actually the case. The linear probes could just learn to predict from some correlated concept/feature. Further evidence would be required that the black-box feature extractor has actually learned the concept that is intervened on.

For interventions and fine-tuning to be effective, the assumption outlined by the reviewer needs to hold. This assumption was explicitly mentioned in the paragraph “Should All Models Be Intervenable?” in Section 3.2 of the initially submitted manuscript (this paragraph has been moved to Appendix B in the revised version).

In our view, if the interventions improve the predictive performance (before or after fine-tuning), this implies that the representations $\boldsymbol{z}$ are informative of the concept variables and, hence, interventions can be performed somewhat effectively. We do not deem it necessary that representations identify concept variables or are disentangled w.r.t. the concepts.

Dear reviewer Emzi,

Given that the end of the discussion period is approaching, we would like to ask if you have any further concerns or questions, particularly as a follow-up to our response? Thank you in advance!

We thank the reviewer for the feedback! Below are our point-by-point responses to the concerns raised.

Limited improvements over CBMs -- In most settings, CBMs seem to outperform the proposed Fine-tuned, I method while providing much greater interpretability (the main exception is in Fig 5).

CBMs are indeed more intervenable than black-box models (even after fine-tuning) on simple benchmarks, such as synthetic bottleneck and confounder (where generative mechanism directly or closely matches the CBM). However, on the synthetic incomplete and chest X-ray datasets, where the set of concepts likely does not fully explain the relationship between $\boldsymbol{x}$ and $y$, fine-tuned black boxes outperform CBMs. These results are not surprising, and we would argue that the proposed approach can be practical and useful for practitioners working with more complex datasets.

Finally, there exist scenarios where training a CBM is impossible or impractical, e.g. when relying on a foundation model [1] or when the concept variables are unknown or too expensive to label at the training time. Thus, we believe there is practical merit in the proposed techniques.

Potential missing baseline: I am not sure if a simple convex combination of the predictions of the CBM and the black-box would outperform the proposed model.

While the suggested baseline might be practical, combining a CBM with a black-box model would lead to interpretations and interventions that are not faithful to the predictions made since the output of the CBM would be augmented with the black-box prediction that is not intervenable or explained in any way. In addition, as mentioned above, our work is motivated by application scenarios in which CBMs are not as practical.

Note: would be nice for the paper to mention the link between intervenability and concept/feature importance - this should be straightforward based on interventions, as this is how many feature importance metrics (e.g. LIME/SHAP) are computed.

Thank you for this suggestion! There indeed exists a relationship between intervenability and feature importance measures.

Drawing a direct relationship with LIME or SHAP is somewhat challenging since these feature importance measures assess the expected change in the model’s output rather than loss. On the other hand, intervenability is very reminiscent of the model reliance as formalised in [2] and explained informally in [3], which is another family of feature importance measures. In particular, model reliance ($MR$) on feature $j$ is given by the ratio of the expected loss of the model $f$ when the feature $j$ is augmented with noise, rendering this feature uninformative of the target variable versus the expected loss under the original data distribution. Similarly, the intervenability (as defined in Equation 2) measures the difference in the expected losses. Suppose the distribution $\pi$ (intervention strategy) is chosen to augment a single concept variable with the noise (as described in [2]). In that case, intervenability may be used to quantify the reliance of $g_{\boldsymbol{\psi}}$ on the individual concepts in $\boldsymbol{\hat{c}}$. We have added a comment on these observations in the revised manuscript in Appendix B.

Would be nice to describe the experimental setup in more detail, e.g. the three synthetic scenarios are only described in previous work.

Note that the synthetic dataset and three different generative mechanisms were described in detail in the appendix of the submitted manuscript. To improve clarity, we have added a brief description of the three scenarios to the main text of the revised manuscript. We have also included a detailed explanation of the intervention strategies in Appendix D.

References

[1] Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023). Foundation models for generalist medical artificial intelligence. Nature, 616(7956), 259-265.

[2] Fisher, A., Rudin, C., & Dominici, F. (2019). All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously. Journal of Machine Learning Research, 20(177), 1-81.

[3] Molnar, C. (2020). Interpretable machine learning. Lulu. com.

I have raised my score to 8 in response to the author's comments. It would still be nice to concretely see scenarios where CBMs fail and the proposed method shows a significant improvement, but the authors have done a decent job supporting that these scenarios exist.

We thank the reviewer for the positive evaluation of our work and detailed feedback! Below are our point-by-point responses to the concerns raised.

The paper proposes interventions on black-box models in CBM style.

Thank you for the neat and accurate summary of our work! We appreciate your effort.

There is no guarantee that a network has learned the desired concepts, i.e. there is a big probability that the probing network can not learn a concept you would want to intervene on.

This limitation is indeed one of the caveats of the proposed intervention recipe. Nevertheless, in the scenario described by the reviewer, the intervenability measure alongside fine-tuning can help practitioners understand that the black-box model is perhaps inadequate for the considered application. In that case, we should resort to (i) in addition, fine-tuning w.r.t. the parameters of the feature extractor $h_{\boldsymbol{\phi}}$ or (ii) using another black-box model.

The method is quite expensive optimizing to get the $z’$ and optimizing for fine-tuning on top of $z’$ can be quite costly.

Interventions require additional optimisation; however, they can be performed on a batch of data points to speed up the inference. By contrast, the fine-tuning is run only once on the validation set (smaller than the training set), and at deployment, only the intervention routine needs to be executed.

Creating a probing network per concept can be costly when we have a large number of concepts, it is not clear if this can scale to hundreds or thousands of concepts.

We would like to emphasise that we use a multivariate probing function in all experiments (for both linear and nonlinear functions), i.e. the probe predicts concept variables in a multitask fashion. Thus, we expect no considerable additional scaling issues when dealing with a large number of concept variables. Please note that, for example, the AwA2 and CUB datasets have 85 and 112 concept variables, respectively, and we observed no scalability problems for these benchmarks. Finally, having thousands of concept variables requires a significant mental effort on the user’s side and, in that setting, the interpretability and utility of a concept-based approach are somewhat limited.

It is not clear which concepts one should intervene on.

Arguably, the expert user should decide which concepts to intervene on, e.g. consider a medical specialist paired up with an ML model predicting the patient’s diagnosis. We represent this decision by the intervention strategy $\pi\left(\boldsymbol{c}' \vert \boldsymbol{x}, \boldsymbol{\hat{c}}, \boldsymbol{c}, \hat{y}, y\right)$. Note that the set of conditioning variables depends on the problem setting. In our experiments, we explore two simple strategies: (i) random-subset, where the set of concept variables is chosen uniformly at random and the values of the chosen variables are replaced with the ground truth, and (ii) uncertainty-based, where the variables are sampled with probabilities proportional to the uncertainty given by $1/\left(\left|\hat{c}_j-0.5\right|+\varepsilon\right)$, where $\varepsilon>0$ is small. Generally, other strategies are viable and can be easily incorporated into our method and experiments, e.g. [1] proposes a few plausible options and conducts a thorough empirical evaluation.

To improve clarity, we have now included a pseudocode description of the considered intervention strategies in the appendix of the revised paper (Appendix D, Algorithms D.1 and D.2).

What if we can never intervene during test time (say we don't have ground-truth concepts or even ground-truth label at that point) it is unclear how this can be useful in that case.

As the intended use case of the proposed method is human interaction with a black-box ML model, neither ground-truth concept nor labels are strictly required since $\boldsymbol{c}’$ is supposed to be provided by the user (these could be either ground-truth, golden-standard, or values reflecting the user’s informed guess). As mentioned, the set of conditioning variables for the strategy $\pi$ may vary depending on the application scenario. In fact, none of the intervention strategies considered in our experiments rely on the ground-truth target label $y$.

On the other hand, if the user cannot provide golden-standard or ground-truth concept values, then a concept-based approach is not helpful for the problem at hand, and the user should resort to a black-box model without interventions.

I would like to thank the authors for their detailed responses to my concerns and questions. I think this work is very interesting and important, and the paper is well-written. My score remains as is.

We thank the reviewer for the feedback! Below are our point-by-point responses to the concerns raised.

I don't really see any use cases where we would want to intervene on standard models instead of just creating a CBM.

There exist use cases when training a CBM is not practical or possible. Firstly, concepts might be unknown to ML practitioners during model development due to the lack of domain knowledge, or it might be too expensive to annotate the training dataset. Secondly, with increased interest in and use of foundation models [1], training dedicated task-specific models becomes less practical and feasible. Hence, there is a need for techniques that allow users to incorporate concept knowledge into representations and interact with predictive models without redesigning and retraining the entire model. This work explores this direction and proposes a practical fine-tuning approach to improve the intervenablity. At the same, the representations could still be used for other downstream tasks, as they remain unaffected by fine-tuning.

Intervention performance on models that weren't finetuned is quite poor, and still requires labeled concept data to learn the classifiers.

It is true that in several datasets, the black-box models are less intervenable than CBMs. However, to the best of our knowledge, this work is the first to explore this research question, and even a somewhat negative finding in this regard is a valuable contribution to the literature, in our view. Finally, our fine-tuning approach mitigates this shortcoming of black-box models.

Regarding the need for concept labels, we fully acknowledge that an annotated validation set is required (rather than an entire training set, as for CBMs). Additionally, our techniques are entirely compatible with the previous approaches for semiautomatic concept discovery, e.g. [2, 3]. Thus, in practice, the need for additional annotation can be alleviated by augmenting our methods with the ideas from the previous works. This research question is, however, beyond the scope of the current manuscript.

The performance of models (even finetuned) is worse than CBM on most datasets, while both require additional training and the same kind of data with dense concept labels, most cases it would be better to just learn a CBM.

While it is true that fine-tuned models are less performant than CBMs on the synthetic data under the bottleneck and confounder mechanisms (and CUB, as observed in our additional experiments), there is a utility in fine-tuning models on the synthetic incomplete and chest X-ray datasets (and AwA, as observed in the updated experiments). The former scenarios are simplistic. In synthetic bottleneck and confounder, generative mechanisms directly or closely match the CBM. We expect CBMs to perform very well on those benchmarks. However, fine-tuned models are expected to perform better in more complex settings (synthetic incomplete and chest X-ray classification), where concept variables may only partially explain the relationship between $y$ and $\boldsymbol{x}$. After fine-tuning for those datasets, interventions allow injecting concept information into the representation without removing additional non-concept-based knowledge utilised by the black box.

Intervening now requires solving an optimization problem making it more costly and harder to understand than original CBM interventions.

Intervening on black boxes indeed requires running the optimisation procedure; however, interventions do not require much time if the representations in the chosen layer are not too high-dimensional and can be performed on batches of instances. Moreover, since we use a multivariate probing function, interventions can be performed on multiple concept variables simultaneously without incurring additional computational complexity. In addition, on the user's side, there is no additional mental effort in understanding the interventions: the user can inspect the probe’s prediction of the concept variables and decide to intervene on a few of those, steering the model’s final prediction. Thus, interventions essentially work analogically to CBMs.

CBMs have many interpretability benefits in addition to intervene-ability, which we lose when using standard architecture, such as predictions being simple functions of interpretable concepts.

We agree that CBMs have many advantages beyond interventions. Nevertheless, as mentioned above, there exist use cases when training a CBM from the beginning is not practical or impossible, especially when working with large neural networks whose representations are utilised in multiple downstream tasks. Thus, the proposed techniques can be an interesting alternative, covering a setting complementary to the typical CBM use case.

Dear reviewer JHNX,

Given that the end of the discussion period is approaching, we would like to ask if you have any further concerns or questions, particularly as a follow-up to our response? Thank you in advance!

Dear reviewer, we would like to double-check if you had a chance to assess the response and changes to the manuscript. Have these arguments and adjustments addressed your concerns?

Thank you for your follow-up!

I think in almost all cases you would be better off with Post-hoc CBM[2] or LF-CBM[3] instead.

Our updated experiments show that training a CBM model post hoc does not result in the same intervention effectiveness as the proposed fine-tuning technique.

While you have compared against a post-hoc CBM like model, I don't think this is a particularly fair comparison, as this is a strange implementation different from [2] and [3]. Most importantly, it is trained in a joint way, while [2][3] and trained sequentially, and joint training is known to lead to poor intervene-ability. For a better comparison you should instead train the CBM sequentially, or better yet use the actual methods of the papers. The most fair comparison would be Post-hoc CBM-h, which should be included as the residual can significantly improve performance.

We acknowledge that this implementation is different from the original paper. As in the rest of the baselines we included in our manuscript, we tried to adjust the architectures and training setup to be as fair as possible, i.e. including a non-linear layer after the bottleneck for enhanced expressivity or probing the original representations directly to the concept set.

We would like to note that the current implementation is quite close to the label-free CBM, except for the joint vs. sequential training. We will investigate the use of sequential training in the camera-ready since the discussion period will end in half an hour.

Regarding the hybrid approach, as mentioned in our previous response, including the sequentially trained residual connection would not affect intervenability since the residual prediction is added to the concept-based prediction at the very end. We would also like to stress that we were interested in investigating the effectiveness of interventions and not in the marginal gains in predictive performance (particularly since w/o interventions, most models perform quite comparably).

Last, my question regarding computational cost was not sufficiently addressed, and I would like to have some actual numbers on how long this takes.

The computational cost varies across datasets and it is dependent on the amount of iterations performed for the optimization. The number of iterations required depends on several factors, e.g. the value of the parameter $\lambda$, the number of concept variables, the dimensionality of $z$, and the number of data points in the batch. Given all these considerations, providing a meaningful answer to this question is quite challenging. In our experience, interventions are considerably less costly than training or fine-tuning and can be performed online on a single GPU.

For example, on the synthetic dataset, with the convergence parameter set to $10^{-6}$ (a very conservative value) and for a batch of 512 data points, the intervention procedure requires approx. 500 to 2500 steps (the number of steps varies depending on the number of given concepts and the value of $\lambda$), which amounts to 0.5 to 2 seconds (for the whole batch) for an untuned black-box model. We use smaller batches and more permissive convergence criteria when fine-tuning, allowing a considerable speedup. In addition, note that the run-time of the intervention procedure is not strictly dependent on the black-box model’s architecture (except for the dimensionality of the layer intervened on).